[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:21 dist-test-slave-s9ct krb5kdc[4904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:21 dist-test-slave-s9ct krb5kdc[4904](info): set up 2 sockets
Sep 29 16:20:21 dist-test-slave-s9ct krb5kdc[4904](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:20:22.510754 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:22 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:22.638701 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33629
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:22.751746 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:22.752060 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:22.752135 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:22.756310 4917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:22.756404 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:22.756440 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:22.756480 4917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:22.756516 4917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:22.762071 4917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:22.763257 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:22.764397 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:22.773331 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:22.773357 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:22.773337 4917 server_base.cc:1034] running on GCE node
W20260929 16:20:22.773337 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:22.774207 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:22.775444 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:22.776600 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790698822776569 us; error 46 us; skew 500 ppm
Sep 29 16:20:22 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698822, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:22.779414 4917 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:22.780360 4917 webserver.cc:460] Webserver started at http://127.4.200.126:45211/ using document root <none> and password file <none>
I20260929 16:20:22.780896 4917 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:22.780954 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:22.781119 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:22.783226 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c"
format_stamp: "Formatted at 2026-09-29 16:20:22 on dist-test-slave-s9ct"
server_key: "c3c7d4f0de178449d687fc5592402706"
server_key_iv: "a08b9653e84f502ca2bc24201df51779"
server_key_version: "encryptionkey@0"
I20260929 16:20:22.783749 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c"
format_stamp: "Formatted at 2026-09-29 16:20:22 on dist-test-slave-s9ct"
server_key: "c3c7d4f0de178449d687fc5592402706"
server_key_iv: "a08b9653e84f502ca2bc24201df51779"
server_key_version: "encryptionkey@0"
I20260929 16:20:22.787544 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:22.789927 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:22.791260 4917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:22.791419 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c"
format_stamp: "Formatted at 2026-09-29 16:20:22 on dist-test-slave-s9ct"
server_key: "c3c7d4f0de178449d687fc5592402706"
server_key_iv: "a08b9653e84f502ca2bc24201df51779"
server_key_version: "encryptionkey@0"
I20260929 16:20:22.791602 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:22.807575 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:22.811059 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:22.811260 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:22.818493 4917 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:33629
I20260929 16:20:22.818519 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:33629 every 8 connection(s)
I20260929 16:20:22.819437 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:22.822073 4985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:22.825356 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 4917
I20260929 16:20:22.825505 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:22.826789 4897 external_mini_cluster.cc:1428] Setting key e9edfedaf43dae63fcadd67fb86a0d2c
I20260929 16:20:22.827581 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: Bootstrap starting.
I20260929 16:20:22.830096 4985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:22.830873 4985 log.cc:824] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:22.832710 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: No bootstrap required, opened a new log
I20260929 16:20:22.837214 4985 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } }
I20260929 16:20:22.837476 4985 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:22.837563 4985 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b802d7de9c46dfaa8a8ec2d93a637c, State: Initialized, Role: FOLLOWER
I20260929 16:20:22.838135 4985 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } }
I20260929 16:20:22.838312 4985 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:22.838423 4985 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:22.838526 4985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:22.839681 4985 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } }
I20260929 16:20:22.840049 4985 leader_election.cc:302] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b802d7de9c46dfaa8a8ec2d93a637c; no voters:
I20260929 16:20:22.840422 4985 leader_election.cc:288] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:22.840494 4990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:22.840678 4990 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [term 1 LEADER]: Becoming Leader. State: Replica: f8b802d7de9c46dfaa8a8ec2d93a637c, State: Running, Role: LEADER
I20260929 16:20:22.840981 4990 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } }
I20260929 16:20:22.841710 4985 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:22.842594 4991 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } } }
I20260929 16:20:22.842799 4991 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:22.843389 4992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b802d7de9c46dfaa8a8ec2d93a637c. Latest consensus state: current_term: 1 leader_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b802d7de9c46dfaa8a8ec2d93a637c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33629 } } }
I20260929 16:20:22.843564 4992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c [sys.catalog]: This master's current role is: LEADER
Sep 29 16:20:22 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
W20260929 16:20:22.846681 5005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:22.846771 5005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:22.846841 4999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:22.847498 4999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:22.853703 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:22.829587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.4.200.126:33629)
0929 16:20:22.830051 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:22.830074 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:22.831227 (+ 1153us) server_negotiation.cc:407] Connection header received
0929 16:20:22.834991 (+ 3764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:22.835013 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:22.835398 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:22.835753 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:22.837903 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:22.839513 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:22.841055 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:22.841602 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:22.841788 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:22.847455 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:22.847462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:22.847477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:22.847511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:22.849846 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:22.850626 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:22.850633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:22.850638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:22.850819 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:22.851244 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:22.851248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:22.851250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:22.851736 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:22.851951 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:22.852637 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 16:20:22.852942 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":142,"threads_started":1}
I20260929 16:20:22.854049 4999 catalog_manager.cc:1347] Generated new cluster ID: 52cf89b457384bf199e3a53aae84d2e4
I20260929 16:20:22.854106 4999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:22.891849 4999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:22.893174 4999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:22.897768 4999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b802d7de9c46dfaa8a8ec2d93a637c: Generated new TSK 0
I20260929 16:20:22.898559 4999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:22.929711 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:23.039477 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:23.039736 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:23.039819 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:23.044209 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:23.044276 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:23.044385 5011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:23.049786 5011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:23.050915 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:23.051971 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:23.061422 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:23.061455 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:23.061461 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:23.061647 5011 server_base.cc:1034] running on GCE node
I20260929 16:20:23.062172 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:23.062840 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:23.064046 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790698823064032 us; error 32 us; skew 500 ppm
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:23.066917 5011 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:23.067704 5011 webserver.cc:460] Webserver started at http://127.4.200.65:46435/ using document root <none> and password file <none>
I20260929 16:20:23.068223 5011 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:23.068279 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:23.068439 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:23.070380 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "d5a39b8b9fee4eb9a9e56f84f55526bc"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "7fb832892bf4770b9a18b0f915f0b58e"
server_key_iv: "7b297ddd59c0dc7765ed2118181d8712"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.070986 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "d5a39b8b9fee4eb9a9e56f84f55526bc"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "7fb832892bf4770b9a18b0f915f0b58e"
server_key_iv: "7b297ddd59c0dc7765ed2118181d8712"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.074681 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:20:23.077064 5026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.078243 5011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:23.078367 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "d5a39b8b9fee4eb9a9e56f84f55526bc"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "7fb832892bf4770b9a18b0f915f0b58e"
server_key_iv: "7b297ddd59c0dc7765ed2118181d8712"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.078524 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:23.091997 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:23.095418 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:23.095636 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:23.096256 5011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:23.097090 5011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:23.097136 5011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.097205 5011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:23.097236 5011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.106889 5011 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44093
I20260929 16:20:23.106957 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44093 every 8 connection(s)
I20260929 16:20:23.107810 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:23.116349 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5011
I20260929 16:20:23.116477 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:23.116750 4897 external_mini_cluster.cc:1428] Setting key 559218a301de5d21b0329ad33fda9fa4
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:23.122171 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.109697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:38447 (local address 127.4.200.126:33629)
0929 16:20:23.109886 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:23.109889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:23.110693 (+ 804us) server_negotiation.cc:407] Connection header received
0929 16:20:23.112056 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:23.112060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:23.112115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:23.112194 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:23.113383 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.113947 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.115132 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.115382 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.115497 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.118204 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:23.118208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:23.118210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:23.118232 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:23.119949 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.120467 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.120472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.120474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.120570 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.120976 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.120979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.120980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.121188 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:23.121299 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:23.121833 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 16:20:23.121990 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:20:23.123095 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.110059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33629 (local address 127.4.200.65:38447)
0929 16:20:23.110544 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:23.110588 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:23.111843 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:23.112366 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:23.112373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:23.112728 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:23.113210 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.113222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.114075 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.114079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.115004 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.115013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.115538 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.115542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.115713 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.116285 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:23.116309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:23.118020 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:23.120073 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.120079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.120092 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.120368 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.120705 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.120708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.120710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.120878 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.121300 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:23.121309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:23.121586 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 16:20:23.121783 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:20:23.122131 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260929 16:20:23.124469 5140 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33629
I20260929 16:20:23.124725 5140 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:23.125216 5140 heartbeater.cc:511] Master 127.4.200.126:33629 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:23.126844 4949 ts_manager.cc:194] Registered new tserver with Master: d5a39b8b9fee4eb9a9e56f84f55526bc (127.4.200.65:44093)
I20260929 16:20:23.128072 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:38447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:23.145123 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:23.257470 5145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:23.257730 5145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:23.257773 5145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:23.261910 5145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:23.261965 5145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:23.262058 5145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:23.267324 5145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:23.268384 5145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:23.269372 5145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:23.279242 5153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:23.279390 5145 server_base.cc:1034] running on GCE node
W20260929 16:20:23.279249 5151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:23.279605 5150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:23.280043 5145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:23.280687 5145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:23.281842 5145 hybrid_clock.cc:648] HybridClock initialized: now 1790698823281829 us; error 35 us; skew 500 ppm
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:23.284680 5145 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:23.285531 5145 webserver.cc:460] Webserver started at http://127.4.200.66:38867/ using document root <none> and password file <none>
I20260929 16:20:23.286082 5145 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:23.286144 5145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:23.286326 5145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:23.288447 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "ecd45be851474e8ab4db1528580ba04c"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "c22ac761788a95808be568b5c854f437"
server_key_iv: "eb5efa5814f12fa05217657a93060524"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.289016 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "ecd45be851474e8ab4db1528580ba04c"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "c22ac761788a95808be568b5c854f437"
server_key_iv: "eb5efa5814f12fa05217657a93060524"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.293241 5145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:20:23.295902 5160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.297151 5145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:23.297272 5145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "ecd45be851474e8ab4db1528580ba04c"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "c22ac761788a95808be568b5c854f437"
server_key_iv: "eb5efa5814f12fa05217657a93060524"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.297387 5145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:23.312860 5145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:23.316196 5145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:23.316399 5145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:23.316991 5145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:23.317802 5145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:23.317847 5145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.317910 5145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:23.317942 5145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.327329 5145 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:45691
I20260929 16:20:23.327353 5273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:45691 every 8 connection(s)
I20260929 16:20:23.328205 5145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:23.331414 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5145
I20260929 16:20:23.331574 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:23.331980 4897 external_mini_cluster.cc:1428] Setting key e800ed4b52a0bfaaa1cf429fe27ede1d
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:20:23.343598 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.329991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:42187 (local address 127.4.200.126:33629)
0929 16:20:23.330127 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:23.330131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:23.331068 (+ 937us) server_negotiation.cc:407] Connection header received
0929 16:20:23.332896 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:23.332899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:23.332945 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:23.333006 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:23.334576 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.335114 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.336403 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.336638 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.336775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.339447 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:23.339451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:23.339454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:23.339491 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:23.341110 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.341739 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.341742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.341744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.341833 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.342255 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.342258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.342262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.342538 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:23.342661 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:23.343266 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 16:20:23.343401 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:23.344550 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.330373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33629 (local address 127.4.200.66:42187)
0929 16:20:23.330905 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:23.330948 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:23.332658 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:23.333180 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:23.333190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:23.333663 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:23.334359 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.334383 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.335247 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.335253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.336252 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.336262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.336820 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.336825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.337003 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.337669 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:23.337687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:23.339262 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:23.341237 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.341243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.341260 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.341546 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.341944 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.341948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.341950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.342154 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.342657 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:23.342687 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:23.342961 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 16:20:23.343161 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:20:23.343537 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260929 16:20:23.345952 5274 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33629
I20260929 16:20:23.346217 5274 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:23.346772 5274 heartbeater.cc:511] Master 127.4.200.126:33629 requested a full tablet report, sending...
I20260929 16:20:23.347919 4949 ts_manager.cc:194] Registered new tserver with Master: ecd45be851474e8ab4db1528580ba04c (127.4.200.66:45691)
I20260929 16:20:23.348819 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:42187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:23.362469 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:23.476840 5279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:23.477077 5279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:23.477119 5279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:23.481307 5279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:23.481376 5279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:23.481488 5279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:23.487013 5279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:23.488116 5279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:23.489138 5279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:23.498525 5284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:23.498535 5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:23.498607 5285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:23.498858 5279 server_base.cc:1034] running on GCE node
I20260929 16:20:23.499330 5279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:23.499934 5279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:23.501106 5279 hybrid_clock.cc:648] HybridClock initialized: now 1790698823501083 us; error 30 us; skew 500 ppm
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:23.503826 5279 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:23.504771 5279 webserver.cc:460] Webserver started at http://127.4.200.67:36767/ using document root <none> and password file <none>
I20260929 16:20:23.505347 5279 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:23.505434 5279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:23.505762 5279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:23.508030 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "a10ee93a91af41b784769f603f4068b2"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "6c76428d781faa608cd39975c90d2516"
server_key_iv: "f930be5dc55b81c8f7501b96ddb85b10"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.508611 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "a10ee93a91af41b784769f603f4068b2"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "6c76428d781faa608cd39975c90d2516"
server_key_iv: "f930be5dc55b81c8f7501b96ddb85b10"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.512370 5279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:20:23.514762 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.515952 5279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:23.516089 5279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "a10ee93a91af41b784769f603f4068b2"
format_stamp: "Formatted at 2026-09-29 16:20:23 on dist-test-slave-s9ct"
server_key: "6c76428d781faa608cd39975c90d2516"
server_key_iv: "f930be5dc55b81c8f7501b96ddb85b10"
server_key_version: "encryptionkey@0"
I20260929 16:20:23.516201 5279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:23.531164 5279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:23.534075 5279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:23.534293 5279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:23.534919 5279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:23.535746 5279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:23.535817 5279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.535913 5279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:23.535964 5279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:23.545322 5279 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:40535
I20260929 16:20:23.545342 5407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:40535 every 8 connection(s)
I20260929 16:20:23.546228 5279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:23.548862 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5279
I20260929 16:20:23.548983 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:23.549288 4897 external_mini_cluster.cc:1428] Setting key 465c68a75235804aa6f9b35fe3270f3c
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:23.560802 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.548134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:46773 (local address 127.4.200.126:33629)
0929 16:20:23.548323 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:23.548328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:23.549557 (+ 1229us) server_negotiation.cc:407] Connection header received
0929 16:20:23.551164 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:23.551167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:23.551223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:23.551319 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:23.552605 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.553140 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.554761 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.554994 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.555102 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.557497 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:23.557505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:23.557507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:23.557532 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:23.558791 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.559300 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.559305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.559307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.559388 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.559730 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.559735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.559737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.559916 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:23.560039 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:23.560515 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 16:20:23.560638 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:20:23.561640 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.548844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33629 (local address 127.4.200.67:46773)
0929 16:20:23.549384 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:23.549447 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:23.550961 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:23.551526 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:23.551536 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:23.551978 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:23.552456 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.552469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.553261 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.553266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.554636 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:23.554645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.555142 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.555147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:23.555314 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.555905 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:23.555922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:23.557326 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:23.558904 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.558910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.558925 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.559194 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.559488 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:23.559491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:23.559493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:23.559633 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:23.560036 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:23.560042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:23.560285 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 16:20:23.560485 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:20:23.560811 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":159,"threads_started":1}
I20260929 16:20:23.562971 5408 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33629
I20260929 16:20:23.563259 5408 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:23.563827 5408 heartbeater.cc:511] Master 127.4.200.126:33629 requested a full tablet report, sending...
I20260929 16:20:23.564814 4949 ts_manager.cc:194] Registered new tserver with Master: a10ee93a91af41b784769f603f4068b2 (127.4.200.67:40535)
I20260929 16:20:23.565402 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:46773
I20260929 16:20:23.575592 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:23.602144 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.592490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.4.200.65:44093)
0929 16:20:23.592810 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:23.592824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:23.592849 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:20:23.592982 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:23.592988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:23.593185 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:23.593319 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:23.594010 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.594971 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.595850 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.596222 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.596313 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.598794 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:23.598803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:23.598817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:23.598843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:23.600481 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.600884 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.600889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.600890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.600963 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.601189 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.601191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.601192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.601508 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:23.601672 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:23.601849 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:20:23.601990 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
W20260929 16:20:23.603192 5054 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54944
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:23.628948 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:23.620607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.4.200.65:44093)
0929 16:20:23.620773 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:23.620778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:23.620910 (+ 132us) server_negotiation.cc:407] Connection header received
0929 16:20:23.621106 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:23.621109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:23.621164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:23.621251 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:23.621941 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.622553 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.623407 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:23.623653 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:23.623763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:23.626181 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:23.626184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:23.626187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:23.626210 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:23.627613 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.627999 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.628002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.628003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.628072 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:23.628289 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:23.628291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:23.628292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:23.628442 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:23.628537 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:23.628719 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:20:23.628816 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:20:23.630000 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5011
I20260929 16:20:23.635749 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5145
I20260929 16:20:23.643322 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5279
I20260929 16:20:23.648713 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 4917
2026-09-29T16:20:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2186 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[5422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[5422](info): set up 2 sockets
Sep 29 16:20:23 dist-test-slave-s9ct krb5kdc[5422](info): commencing operation
krb5kdc: starting...
W20260929 16:20:24.690104 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:24 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:24.812885 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43113
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:43113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:24.925453 5435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:24.925808 5435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:24.925911 5435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:24.930194 5435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:24.930303 5435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:24.930341 5435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:24.930377 5435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:24.930439 5435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:24.936220 5435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:43113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:24.938060 5435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:24.940842 5435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:24.950310 5440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:24.950320 5443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:24.950331 5441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:24.950598 5435 server_base.cc:1034] running on GCE node
I20260929 16:20:24.951294 5435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:24.952425 5435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:24.953657 5435 hybrid_clock.cc:648] HybridClock initialized: now 1790698824953623 us; error 48 us; skew 500 ppm
Sep 29 16:20:24 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698824, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:24.963548 5435 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:24.964671 5435 webserver.cc:460] Webserver started at http://127.4.200.126:35681/ using document root <none> and password file <none>
I20260929 16:20:24.965276 5435 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:24.965375 5435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:24.965613 5435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:24.967810 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "ff47df3b5b344701bbc7578e64e6ae0c"
format_stamp: "Formatted at 2026-09-29 16:20:24 on dist-test-slave-s9ct"
server_key: "0b35c40f100f410dab571fd116b53432"
server_key_iv: "eef4493d2fb785f48b396b18b023737e"
server_key_version: "encryptionkey@0"
I20260929 16:20:24.968400 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "ff47df3b5b344701bbc7578e64e6ae0c"
format_stamp: "Formatted at 2026-09-29 16:20:24 on dist-test-slave-s9ct"
server_key: "0b35c40f100f410dab571fd116b53432"
server_key_iv: "eef4493d2fb785f48b396b18b023737e"
server_key_version: "encryptionkey@0"
I20260929 16:20:24.972344 5435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:20:24.975016 5450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:24.976467 5435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:24.976606 5435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ff47df3b5b344701bbc7578e64e6ae0c"
format_stamp: "Formatted at 2026-09-29 16:20:24 on dist-test-slave-s9ct"
server_key: "0b35c40f100f410dab571fd116b53432"
server_key_iv: "eef4493d2fb785f48b396b18b023737e"
server_key_version: "encryptionkey@0"
I20260929 16:20:24.976734 5435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:25.013624 5435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:25.016889 5435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:25.017112 5435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:25.024402 5435 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:43113
I20260929 16:20:25.024430 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:43113 every 8 connection(s)
I20260929 16:20:25.025408 5435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:25.028249 5503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:25.029886 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5435
I20260929 16:20:25.029983 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:25.030244 4897 external_mini_cluster.cc:1428] Setting key 211fee253a256b27817d35fb3c9f1e18
I20260929 16:20:25.035010 5503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c: Bootstrap starting.
I20260929 16:20:25.037761 5503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:25.038573 5503 log.cc:824] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:25.040570 5503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c: No bootstrap required, opened a new log
I20260929 16:20:25.045429 5503 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } }
I20260929 16:20:25.045722 5503 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:25.045786 5503 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff47df3b5b344701bbc7578e64e6ae0c, State: Initialized, Role: FOLLOWER
I20260929 16:20:25.046288 5503 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } }
I20260929 16:20:25.046480 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.031742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.4.200.126:43113)
0929 16:20:25.032201 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.032217 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.032252 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:20:25.032939 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.032955 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.033349 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.033770 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:25.035061 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.036762 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.037629 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.038174 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.038347 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.040926 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:25.040934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:25.040953 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:25.040984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:25.043086 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.043569 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.043575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.043581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.043719 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.044053 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.044057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.044060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.044599 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:25.044801 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.045074 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.045378 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260929 16:20:25.046478 5503 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:25.046588 5503 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:25.046675 5503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:25.047859 5503 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } }
I20260929 16:20:25.048226 5503 leader_election.cc:302] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff47df3b5b344701bbc7578e64e6ae0c; no voters:
I20260929 16:20:25.048552 5503 leader_election.cc:288] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:25.048663 5508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:25.048904 5508 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [term 1 LEADER]: Becoming Leader. State: Replica: ff47df3b5b344701bbc7578e64e6ae0c, State: Running, Role: LEADER
I20260929 16:20:25.049222 5508 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } }
I20260929 16:20:25.049860 5503 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:25.050997 5510 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } } }
I20260929 16:20:25.051146 5510 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:25.051440 5509 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff47df3b5b344701bbc7578e64e6ae0c. Latest consensus state: current_term: 1 leader_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff47df3b5b344701bbc7578e64e6ae0c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43113 } } }
I20260929 16:20:25.051525 5509 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:25.051546 5517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:25.054869 5517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:25.060633 5517 catalog_manager.cc:1347] Generated new cluster ID: 3589f08bd52f4a89803cd2229c50bd05
I20260929 16:20:25.060719 5517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:25.085875 5517 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:25.086973 5517 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:25.099789 5517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff47df3b5b344701bbc7578e64e6ae0c: Generated new TSK 0
I20260929 16:20:25.100714 5517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:25.138854 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:25.247819 5529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:25.248073 5529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:25.248116 5529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:25.252616 5529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:25.252673 5529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:25.252754 5529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:25.257969 5529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:25.259018 5529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:25.260023 5529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:25.269374 5534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:25.269464 5529 server_base.cc:1034] running on GCE node
W20260929 16:20:25.269426 5535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:25.269323 5537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:25.269994 5529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:25.270704 5529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:25.271891 5529 hybrid_clock.cc:648] HybridClock initialized: now 1790698825271875 us; error 27 us; skew 500 ppm
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:25.274617 5529 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:25.275413 5529 webserver.cc:460] Webserver started at http://127.4.200.65:38651/ using document root <none> and password file <none>
I20260929 16:20:25.275985 5529 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:25.276226 5529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:25.276484 5529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:25.278661 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "28b72e0570324e23a7830c9877953c26"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "e2b3919274c803ae4df6e20150f7e286"
server_key_iv: "be93a4ea8b6bb7d035d8c8d1f7cf406f"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.279209 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "28b72e0570324e23a7830c9877953c26"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "e2b3919274c803ae4df6e20150f7e286"
server_key_iv: "be93a4ea8b6bb7d035d8c8d1f7cf406f"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.283043 5529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:20:25.285480 5544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.286772 5529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:25.286914 5529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "28b72e0570324e23a7830c9877953c26"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "e2b3919274c803ae4df6e20150f7e286"
server_key_iv: "be93a4ea8b6bb7d035d8c8d1f7cf406f"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.287024 5529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:25.305552 5529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:25.308470 5529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:25.308662 5529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:25.309242 5529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:25.310032 5529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:25.310101 5529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.310169 5529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:25.310220 5529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.319020 5529 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:33927
I20260929 16:20:25.319039 5657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:33927 every 8 connection(s)
I20260929 16:20:25.319911 5529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:25.324779 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5529
I20260929 16:20:25.324908 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:25.325171 4897 external_mini_cluster.cc:1428] Setting key c899bbb85ee2298467dcc82b7addc8ac
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:25.334830 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.321727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:42125 (local address 127.4.200.126:43113)
0929 16:20:25.321903 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.321907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.322799 (+ 892us) server_negotiation.cc:407] Connection header received
0929 16:20:25.324175 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.324178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.324239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.324339 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:25.325652 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.326290 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.327548 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.327984 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.328108 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.331220 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:25.331223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:25.331225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:25.331248 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:25.332738 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.333273 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.333276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.333277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.333349 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.333697 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.333701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.333702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.333880 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:25.333999 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.334513 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.334663 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:20:25.335650 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.322099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43113 (local address 127.4.200.65:42125)
0929 16:20:25.322645 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.322689 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.323946 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.324520 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.324528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.324878 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:25.325476 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.325491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.326471 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.326474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.327417 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.327427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.328182 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.328188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.328417 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.329054 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:25.329083 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:25.331043 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:25.332865 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.332871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.332887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.333152 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.333444 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.333447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.333456 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.333595 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.333999 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:25.334007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:25.334250 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.334467 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.334803 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":113,"threads_started":1}
I20260929 16:20:25.336936 5658 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:20:25.337195 5658 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:25.337683 5658 heartbeater.cc:511] Master 127.4.200.126:43113 requested a full tablet report, sending...
I20260929 16:20:25.339192 5467 ts_manager.cc:194] Registered new tserver with Master: 28b72e0570324e23a7830c9877953c26 (127.4.200.65:33927)
I20260929 16:20:25.340786 5467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:42125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:25.353554 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:25.462268 5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:25.462580 5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:25.462658 5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:25.466744 5663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:25.466797 5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:25.466899 5663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:25.472240 5663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:25.473246 5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:25.474242 5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:25.484125 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:25.484110 5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:25.484228 5663 server_base.cc:1034] running on GCE node
W20260929 16:20:25.484110 5671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:25.484876 5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:25.485519 5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:25.486702 5663 hybrid_clock.cc:648] HybridClock initialized: now 1790698825486686 us; error 30 us; skew 500 ppm
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:25.489396 5663 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:25.490280 5663 webserver.cc:460] Webserver started at http://127.4.200.66:46361/ using document root <none> and password file <none>
I20260929 16:20:25.490958 5663 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:25.491045 5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:25.491271 5663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:25.493283 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "9dedb61252a24c5193d72b02759abfa3"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "8d91af14c80e8522404d638f6c4cea72"
server_key_iv: "8164166d515cabb32e054dc4925bb8e6"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.493848 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "9dedb61252a24c5193d72b02759abfa3"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "8d91af14c80e8522404d638f6c4cea72"
server_key_iv: "8164166d515cabb32e054dc4925bb8e6"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.497545 5663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:20:25.500010 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.501268 5663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:25.501412 5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "9dedb61252a24c5193d72b02759abfa3"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "8d91af14c80e8522404d638f6c4cea72"
server_key_iv: "8164166d515cabb32e054dc4925bb8e6"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.501526 5663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:25.513716 5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:25.516541 5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:25.516744 5663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:25.517311 5663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:25.518105 5663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:25.518179 5663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.518261 5663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:25.518316 5663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.527473 5663 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:43905
I20260929 16:20:25.527498 5791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:43905 every 8 connection(s)
I20260929 16:20:25.528354 5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:25.529520 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5663
I20260929 16:20:25.529659 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:25.530019 4897 external_mini_cluster.cc:1428] Setting key a7bb853ee224af086a6749a54666c058
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:25.543670 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.530337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:34867 (local address 127.4.200.126:43113)
0929 16:20:25.530484 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.530493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.531494 (+ 1001us) server_negotiation.cc:407] Connection header received
0929 16:20:25.532866 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.532872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.532931 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.533028 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:25.534655 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.535537 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.536852 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.537285 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.537421 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.540037 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:25.540040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:25.540042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:25.540065 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:25.541479 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.542031 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.542033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.542034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.542105 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.542519 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.542525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.542530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.542741 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:25.542821 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.543443 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.543549 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:25.544628 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.530853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43113 (local address 127.4.200.66:34867)
0929 16:20:25.531331 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.531379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.532651 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.533252 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.533263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.533764 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:25.534487 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.534504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.535680 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.535683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.536695 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.536706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.537470 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.537473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.537622 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.538285 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:25.538302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:25.539874 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:25.541631 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.541638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.541654 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.541937 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.542204 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.542207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.542209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.542339 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.542853 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:25.542863 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:25.543190 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.543396 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.543761 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":152,"threads_started":1}
I20260929 16:20:25.545838 5792 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43113
I20260929 16:20:25.546126 5792 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:25.546684 5792 heartbeater.cc:511] Master 127.4.200.126:43113 requested a full tablet report, sending...
I20260929 16:20:25.547854 5467 ts_manager.cc:194] Registered new tserver with Master: 9dedb61252a24c5193d72b02759abfa3 (127.4.200.66:43905)
I20260929 16:20:25.548687 5467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:34867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:25.557461 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:25.666837 5797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:25.667110 5797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:25.667156 5797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:25.671182 5797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:25.671241 5797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:25.671343 5797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:25.676790 5797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:25.677994 5797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:25.679041 5797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:25.688592 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:25.688616 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:25.688633 5805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:25.688660 5797 server_base.cc:1034] running on GCE node
I20260929 16:20:25.689258 5797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:25.689862 5797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:25.691035 5797 hybrid_clock.cc:648] HybridClock initialized: now 1790698825691022 us; error 29 us; skew 500 ppm
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:25.693910 5797 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:25.694773 5797 webserver.cc:460] Webserver started at http://127.4.200.67:40475/ using document root <none> and password file <none>
I20260929 16:20:25.695294 5797 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:25.695351 5797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:25.695524 5797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:25.697507 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "c322d5ca00ce41d28b31301bd8c4fa79"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "d059370f6789d4458752909cbc9693bc"
server_key_iv: "d4af72e02dc5eef685ab2f3d5453d6cb"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.698027 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "c322d5ca00ce41d28b31301bd8c4fa79"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "d059370f6789d4458752909cbc9693bc"
server_key_iv: "d4af72e02dc5eef685ab2f3d5453d6cb"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.701766 5797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:20:25.704495 5812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.705912 5797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:20:25.706033 5797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "c322d5ca00ce41d28b31301bd8c4fa79"
format_stamp: "Formatted at 2026-09-29 16:20:25 on dist-test-slave-s9ct"
server_key: "d059370f6789d4458752909cbc9693bc"
server_key_iv: "d4af72e02dc5eef685ab2f3d5453d6cb"
server_key_version: "encryptionkey@0"
I20260929 16:20:25.706146 5797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:25.734377 5797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:25.737771 5797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:25.737985 5797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:25.738634 5797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:25.739462 5797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:25.739508 5797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.739547 5797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:25.739598 5797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:25.748522 5797 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:34279
I20260929 16:20:25.748544 5925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:34279 every 8 connection(s)
I20260929 16:20:25.749419 5797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:25.753962 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5797
I20260929 16:20:25.754086 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:25.754457 4897 external_mini_cluster.cc:1428] Setting key fa731d254da3fe6fad78bab696bcb996
Sep 29 16:20:25 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:25.763682 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.751363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:55911 (local address 127.4.200.126:43113)
0929 16:20:25.751564 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.751568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.752421 (+ 853us) server_negotiation.cc:407] Connection header received
0929 16:20:25.754149 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.754155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.754211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.754293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:25.755779 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.756299 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.757444 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.757670 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.757780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.760312 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:25.760314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:25.760316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:25.760363 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:25.761774 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.762257 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.762259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.762261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.762352 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.762728 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.762730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.762731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.762934 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:25.763012 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.763452 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.763556 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:20:25.764521 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.751794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43113 (local address 127.4.200.67:55911)
0929 16:20:25.752257 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.752301 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.753563 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.754557 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.754566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.755056 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:25.755632 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.755667 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.756435 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.756438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.757319 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.757329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.757815 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.757819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.758013 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.758662 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:25.758678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:25.760154 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:25.761889 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.761894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.761906 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.762170 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.762497 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:25.762499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:25.762500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:25.762642 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:25.763021 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:25.763025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:25.763253 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.763407 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.763666 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260929 16:20:25.765698 5926 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43113
I20260929 16:20:25.765944 5926 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:25.766485 5926 heartbeater.cc:511] Master 127.4.200.126:43113 requested a full tablet report, sending...
I20260929 16:20:25.767628 5467 ts_manager.cc:194] Registered new tserver with Master: c322d5ca00ce41d28b31301bd8c4fa79 (127.4.200.67:34279)
I20260929 16:20:25.768270 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:20:25.768337 5467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:55911
I20260929 16:20:25.778955 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.771794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.4.200.126:43113)
0929 16:20:25.771945 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.771949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.772117 (+ 168us) server_negotiation.cc:407] Connection header received
0929 16:20:25.772257 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.772260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.772302 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.772392 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:25.773090 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.773598 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.774422 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.774657 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.774766 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.776113 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:25.776117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:25.776120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:25.776144 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:25.777339 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.777749 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.777755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.777757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.777866 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:25.778156 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:25.778159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:25.778160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:25.778382 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:25.778554 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.778744 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.778839 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:20:25.783516 5467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:25.786541 5467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:25.800726 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.794925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:34279 (local address 127.0.0.1:54486)
0929 16:20:25.795531 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.795544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.795632 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.796329 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.796336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.796362 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:25.796513 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.796521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.797932 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.797938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.799370 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.799379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.800182 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.800186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.800296 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.800332 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.800440 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.800561 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":108,"thread_start_us":119,"threads_started":1}
I20260929 16:20:25.800763 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.795548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:43905 (local address 127.0.0.1:50138)
0929 16:20:25.795985 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.795999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.796087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.796780 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.796783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.796799 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:25.796892 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.796898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.798002 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.798005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.799376 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.799387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.800446 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.800449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.800534 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.800553 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.800605 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.800659 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":55,"thread_start_us":96,"threads_started":1}
I20260929 16:20:25.800897 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.795935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.4.200.66:43905)
0929 16:20:25.796342 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.796353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.796374 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:20:25.796441 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.796446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.796642 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.796799 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:25.797008 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.797885 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.799519 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.800335 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.800451 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.800587 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.800651 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.800741 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260929 16:20:25.801671 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.794784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:33927 (local address 127.0.0.1:50446)
0929 16:20:25.795308 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.795355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.795492 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.797138 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.797141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.797159 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:25.797246 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.797252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.798809 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.798813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.800169 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.800180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.801267 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.801271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:25.801357 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.801391 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:20:25.801471 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:20:25.801542 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":113,"threads_started":1}
I20260929 16:20:25.801733 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.795173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.4.200.65:33927)
0929 16:20:25.796714 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.796728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.796749 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:20:25.796819 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.796824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.797002 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.797152 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:25.797378 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.798533 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.800327 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.801142 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.801267 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.801399 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.801499 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.801588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434,"thread_start_us":102,"threads_started":1}
I20260929 16:20:25.801780 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.795508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.4.200.67:34279)
0929 16:20:25.795791 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.795802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.795825 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:20:25.795902 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.795908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.796124 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.796323 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:25.796636 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.797793 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.799493 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.800060 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.800184 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:25.800333 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:25.801535 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0929 16:20:25.801637 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260929 16:20:25.804267 5726 tablet_service.cc:1467] Processing CreateTablet for tablet 6b069ed3269f4729b94569ab656b134a (DEFAULT_TABLE table=test-table [id=96868b302c48436a9f2ed0c02981741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:25.804345 5860 tablet_service.cc:1467] Processing CreateTablet for tablet 6b069ed3269f4729b94569ab656b134a (DEFAULT_TABLE table=test-table [id=96868b302c48436a9f2ed0c02981741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:25.805351 5860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b069ed3269f4729b94569ab656b134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:25.805361 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b069ed3269f4729b94569ab656b134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:25.805284 5592 tablet_service.cc:1467] Processing CreateTablet for tablet 6b069ed3269f4729b94569ab656b134a (DEFAULT_TABLE table=test-table [id=96868b302c48436a9f2ed0c02981741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:25.806191 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b069ed3269f4729b94569ab656b134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:25.812580 5943 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Bootstrap starting.
I20260929 16:20:25.813191 5944 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Bootstrap starting.
I20260929 16:20:25.815548 5944 tablet_bootstrap.cc:654] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:25.816039 5945 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Bootstrap starting.
I20260929 16:20:25.816141 5943 tablet_bootstrap.cc:654] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:25.816387 5944 log.cc:824] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:25.817807 5943 log.cc:824] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:25.818614 5945 tablet_bootstrap.cc:654] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:25.818652 5944 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: No bootstrap required, opened a new log
I20260929 16:20:25.818856 5944 ts_tablet_manager.cc:1397] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:20:25.819831 5945 log.cc:824] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:25.819905 5943 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: No bootstrap required, opened a new log
I20260929 16:20:25.820080 5943 ts_tablet_manager.cc:1397] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:20:25.821785 5945 tablet_bootstrap.cc:492] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: No bootstrap required, opened a new log
I20260929 16:20:25.821969 5945 ts_tablet_manager.cc:1397] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:20:25.822386 5944 raft_consensus.cc:348] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.822692 5944 raft_consensus.cc:374] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:25.822786 5944 raft_consensus.cc:729] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dedb61252a24c5193d72b02759abfa3, State: Initialized, Role: FOLLOWER
I20260929 16:20:25.823272 5944 consensus_queue.cc:260] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.823303 5943 raft_consensus.cc:348] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.823560 5943 raft_consensus.cc:374] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:25.823658 5943 raft_consensus.cc:729] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c322d5ca00ce41d28b31301bd8c4fa79, State: Initialized, Role: FOLLOWER
I20260929 16:20:25.823982 5792 heartbeater.cc:503] Master 127.4.200.126:43113 was elected leader, sending a full tablet report...
I20260929 16:20:25.824143 5943 consensus_queue.cc:260] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.825089 5943 ts_tablet_manager.cc:1428] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:20:25.825182 5944 ts_tablet_manager.cc:1428] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:20:25.825426 5926 heartbeater.cc:503] Master 127.4.200.126:43113 was elected leader, sending a full tablet report...
I20260929 16:20:25.825634 5945 raft_consensus.cc:348] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.825848 5945 raft_consensus.cc:374] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:25.825908 5945 raft_consensus.cc:729] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b72e0570324e23a7830c9877953c26, State: Initialized, Role: FOLLOWER
I20260929 16:20:25.826301 5945 consensus_queue.cc:260] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.827015 5658 heartbeater.cc:503] Master 127.4.200.126:43113 was elected leader, sending a full tablet report...
I20260929 16:20:25.827270 5945 ts_tablet_manager.cc:1428] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:20:25.995086 5951 raft_consensus.cc:482] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:25.995291 5951 raft_consensus.cc:504] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:25.996392 5951 leader_election.cc:288] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c322d5ca00ce41d28b31301bd8c4fa79 (127.4.200.67:34279), 9dedb61252a24c5193d72b02759abfa3 (127.4.200.66:43905)
W20260929 16:20:26.000424 5927 tablet.cc:2369] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:26.001582 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.996956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56293 (local address 127.4.200.67:34279)
0929 16:20:25.997105 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.997110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.997187 (+ 77us) server_negotiation.cc:407] Connection header received
0929 16:20:25.997378 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.997382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.997444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.997549 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:25.997859 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.998673 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.000293 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.000937 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.001057 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.001109 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:26.001381 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:20:26.001454 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:20:26.001582 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.997307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:34633 (local address 127.4.200.66:43905)
0929 16:20:25.997430 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:25.997434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:25.997711 (+ 277us) server_negotiation.cc:407] Connection header received
0929 16:20:25.997811 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:25.997816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:25.997876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:25.997963 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:25.998188 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.998911 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.000290 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.000922 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.001057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.001109 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:26.001381 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:20:26.001454 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:20:26.001634 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.996711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:34279 (local address 127.4.200.65:56293)
0929 16:20:25.997083 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.997101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.997235 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.997576 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.997579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.997600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:25.997742 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.997750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.998800 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.998803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:26.000126 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:26.000137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.001062 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.001066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:26.001226 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.001276 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:20:26.001352 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:20:26.001456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":86,"thread_start_us":143,"threads_started":1}
I20260929 16:20:26.001632 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:25.997195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:43905 (local address 127.4.200.65:34633)
0929 16:20:25.997600 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:25.997613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:25.997702 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:25.997974 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:25.997976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:25.997990 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:25.998070 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:25.998075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:25.999017 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:25.999020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:26.000155 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:26.000162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.001062 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.001066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:26.001231 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.001269 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:20:26.001352 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:20:26.001461 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":78,"thread_start_us":153,"threads_started":1}
I20260929 16:20:26.002350 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b069ed3269f4729b94569ab656b134a" candidate_uuid: "28b72e0570324e23a7830c9877953c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" is_pre_election: true
I20260929 16:20:26.002354 5746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b069ed3269f4729b94569ab656b134a" candidate_uuid: "28b72e0570324e23a7830c9877953c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dedb61252a24c5193d72b02759abfa3" is_pre_election: true
I20260929 16:20:26.002707 5880 raft_consensus.cc:2454] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b72e0570324e23a7830c9877953c26 in term 0.
I20260929 16:20:26.002714 5746 raft_consensus.cc:2454] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b72e0570324e23a7830c9877953c26 in term 0.
I20260929 16:20:26.003208 5545 leader_election.cc:302] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b72e0570324e23a7830c9877953c26, c322d5ca00ce41d28b31301bd8c4fa79; no voters:
I20260929 16:20:26.003494 5951 raft_consensus.cc:2781] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:26.003593 5951 raft_consensus.cc:482] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:26.003640 5951 raft_consensus.cc:3037] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:26.004706 5951 raft_consensus.cc:504] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:26.005080 5951 leader_election.cc:288] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [CANDIDATE]: Term 1 election: Requested vote from peers c322d5ca00ce41d28b31301bd8c4fa79 (127.4.200.67:34279), 9dedb61252a24c5193d72b02759abfa3 (127.4.200.66:43905)
I20260929 16:20:26.005424 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b069ed3269f4729b94569ab656b134a" candidate_uuid: "28b72e0570324e23a7830c9877953c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c322d5ca00ce41d28b31301bd8c4fa79"
I20260929 16:20:26.005466 5746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b069ed3269f4729b94569ab656b134a" candidate_uuid: "28b72e0570324e23a7830c9877953c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dedb61252a24c5193d72b02759abfa3"
I20260929 16:20:26.005570 5880 raft_consensus.cc:3037] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:26.005599 5746 raft_consensus.cc:3037] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:26.006877 5746 raft_consensus.cc:2454] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b72e0570324e23a7830c9877953c26 in term 1.
I20260929 16:20:26.007133 5880 raft_consensus.cc:2454] T 6b069ed3269f4729b94569ab656b134a P c322d5ca00ce41d28b31301bd8c4fa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b72e0570324e23a7830c9877953c26 in term 1.
I20260929 16:20:26.007200 5545 leader_election.cc:302] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b72e0570324e23a7830c9877953c26, 9dedb61252a24c5193d72b02759abfa3; no voters:
I20260929 16:20:26.007408 5951 raft_consensus.cc:2781] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:26.007647 5951 raft_consensus.cc:686] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [term 1 LEADER]: Becoming Leader. State: Replica: 28b72e0570324e23a7830c9877953c26, State: Running, Role: LEADER
I20260929 16:20:26.007982 5951 consensus_queue.cc:237] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } }
I20260929 16:20:26.011409 5464 catalog_manager.cc:5495] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b72e0570324e23a7830c9877953c26 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "28b72e0570324e23a7830c9877953c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b72e0570324e23a7830c9877953c26" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c322d5ca00ce41d28b31301bd8c4fa79" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dedb61252a24c5193d72b02759abfa3" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43905 } health_report { overall_health: UNKNOWN } } }
W20260929 16:20:26.029464 5793 tablet.cc:2369] T 6b069ed3269f4729b94569ab656b134a P 9dedb61252a24c5193d72b02759abfa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:26.038862 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:26.028727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.4.200.65:33927)
0929 16:20:26.028862 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:26.028866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:26.029042 (+ 176us) server_negotiation.cc:407] Connection header received
0929 16:20:26.029193 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:26.029197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:26.029248 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:26.029342 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:26.030180 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.030737 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.031882 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.032079 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.032152 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.034944 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:26.034955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:26.034966 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:26.034990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:26.036659 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.037164 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.037170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.037201 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.037336 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.037656 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.037659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.037661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.038003 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:26.038220 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:26.038562 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:20:26.038703 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:26.065661 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:26.057182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.4.200.65:33927)
0929 16:20:26.057348 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:26.057353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:26.057486 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:20:26.057659 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:26.057667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:26.057710 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:26.057799 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:26.058522 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.059052 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.059915 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.060147 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.060260 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.062712 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:26.062715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:26.062717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:26.062743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:26.063967 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.064391 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.064395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.064397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.064485 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.064759 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.064762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.064764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.064932 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:26.065090 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:26.065364 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:20:26.065498 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:20:26.071213 5659 tablet.cc:2369] T 6b069ed3269f4729b94569ab656b134a P 28b72e0570324e23a7830c9877953c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:26.092306 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:26.083915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.4.200.65:33927)
0929 16:20:26.084085 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:26.084090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:26.084228 (+ 138us) server_negotiation.cc:407] Connection header received
0929 16:20:26.084422 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:26.084425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:26.084469 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:26.084562 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:26.085286 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.085786 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.086679 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:26.086905 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:26.086997 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:26.089387 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:26.089403 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:26.089405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:26.089431 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:26.090903 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.091348 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.091350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.091351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.091439 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:26.091650 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:26.091652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:26.091653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:26.091803 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:26.091914 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:26.092090 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:20:26.092185 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:20:26.093703 5572 tablet_service.cc:2908] Rejecting scan request for tablet 6b069ed3269f4729b94569ab656b134a: Uninitialized: clean time has not yet been initialized
I20260929 16:20:26.095927 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5529
I20260929 16:20:26.101902 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5663
I20260929 16:20:26.107409 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5797
I20260929 16:20:26.112934 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5435
2026-09-29T16:20:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2464 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5968](info): set up 2 sockets
Sep 29 16:20:26 dist-test-slave-s9ct krb5kdc[5968](info): commencing operation
krb5kdc: starting...
W20260929 16:20:27.158177 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:27.281194 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45693
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:27.393963 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:27.394210 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:27.394250 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:27.398200 5981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:27.398250 5981 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:20:27.398267 5981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:27.398317 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:27.398348 5981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:27.398406 5981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:27.403597 5981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45693
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:27.404609 5981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:27.405616 5981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:27.414016 5986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:27.414032 5989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:27.414026 5987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:27.414072 5981 server_base.cc:1034] running on GCE node
I20260929 16:20:27.414878 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:27.415943 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:27.417109 5981 hybrid_clock.cc:648] HybridClock initialized: now 1790698827417097 us; error 30 us; skew 500 ppm
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:27.419953 5981 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:27.420908 5981 webserver.cc:460] Webserver started at http://127.4.200.126:34035/ using document root <none> and password file <none>
I20260929 16:20:27.421471 5981 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:27.421532 5981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:27.421746 5981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:27.423875 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "6d01a44874654b0895291c86f587a023"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "75e7fcc7bc2abdfad9ef0c0a7782c5df"
server_key_iv: "28c2b569916421d1e462a35e02f28539"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.424472 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "6d01a44874654b0895291c86f587a023"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "75e7fcc7bc2abdfad9ef0c0a7782c5df"
server_key_iv: "28c2b569916421d1e462a35e02f28539"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.428324 5981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 16:20:27.430804 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:27.431991 5981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:27.432116 5981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "6d01a44874654b0895291c86f587a023"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "75e7fcc7bc2abdfad9ef0c0a7782c5df"
server_key_iv: "28c2b569916421d1e462a35e02f28539"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.432219 5981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:27.450062 5981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:27.452817 5981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:27.453013 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:27.461043 5981 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45693
I20260929 16:20:27.461066 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45693 every 8 connection(s)
I20260929 16:20:27.462016 5981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:27.464725 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:27.467949 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 5981
I20260929 16:20:27.468071 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:27.468345 4897 external_mini_cluster.cc:1428] Setting key 5fcdd6ed960097d0f3c526205da8eff5
I20260929 16:20:27.470824 6059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Bootstrap starting.
I20260929 16:20:27.472954 6059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:27.473696 6059 log.cc:824] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:27.475606 6059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: No bootstrap required, opened a new log
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:27.479575 6059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } }
I20260929 16:20:27.479823 6059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:27.479910 6059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d01a44874654b0895291c86f587a023, State: Initialized, Role: FOLLOWER
I20260929 16:20:27.480391 6059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } }
I20260929 16:20:27.480543 6059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:27.480614 6059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:27.480782 6059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:27.481691 6059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } }
I20260929 16:20:27.482084 6059 leader_election.cc:302] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d01a44874654b0895291c86f587a023; no voters:
I20260929 16:20:27.482439 6059 leader_election.cc:288] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:27.482510 6064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:27.482736 6064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [term 1 LEADER]: Becoming Leader. State: Replica: 6d01a44874654b0895291c86f587a023, State: Running, Role: LEADER
I20260929 16:20:27.483036 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.469779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.4.200.126:45693)
0929 16:20:27.470223 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:27.470238 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:27.470273 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:20:27.471011 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:27.471029 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:27.471354 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:27.471673 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:27.472456 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.473522 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.474465 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.474947 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.475114 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.477857 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:27.477862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:27.477875 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:27.477901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:27.479646 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.480164 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.480170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.480174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.480279 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.480574 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.480580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.480584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.481092 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:27.481303 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:27.481575 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:20:27.482045 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":132,"threads_started":1}
I20260929 16:20:27.483103 6064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } }
I20260929 16:20:27.483742 6059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:27.484329 6066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d01a44874654b0895291c86f587a023. Latest consensus state: current_term: 1 leader_uuid: "6d01a44874654b0895291c86f587a023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } } }
I20260929 16:20:27.484467 6066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:27.486301 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d01a44874654b0895291c86f587a023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d01a44874654b0895291c86f587a023" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45693 } } }
I20260929 16:20:27.486443 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023 [sys.catalog]: This master's current role is: LEADER
W20260929 16:20:27.488663 6079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:27.488737 6079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:27.488811 6070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:27.489514 6070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:27.494874 6070 catalog_manager.cc:1347] Generated new cluster ID: f7c6211afdf64d5db329ae2840d49971
I20260929 16:20:27.494936 6070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:27.503693 6070 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:27.504581 6070 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:27.508334 6070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Generated new TSK 0
I20260929 16:20:27.508983 6070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:27.537173 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45693
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:27.647905 6085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:27.648167 6085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:27.648242 6085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:27.652196 6085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:27.652274 6085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:27.652349 6085 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:27.652427 6085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:27.657438 6085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:27.658466 6085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:27.659459 6085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:27.668658 6090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:27.668723 6093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:27.668883 6085 server_base.cc:1034] running on GCE node
W20260929 16:20:27.668679 6091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:27.669386 6085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:27.669991 6085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:27.671173 6085 hybrid_clock.cc:648] HybridClock initialized: now 1790698827671161 us; error 24 us; skew 500 ppm
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:27.673843 6085 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:27.674757 6085 webserver.cc:460] Webserver started at http://127.4.200.65:45553/ using document root <none> and password file <none>
I20260929 16:20:27.675319 6085 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:27.675407 6085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:27.675634 6085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:27.677645 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "3bfdee97942f4ae2a17c96ace810b662"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "29884a91dcfb8e1bf8db7e0dac7a4273"
server_key_iv: "95adb57a8fbfe199cade22d0505b86a5"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.678200 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "3bfdee97942f4ae2a17c96ace810b662"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "29884a91dcfb8e1bf8db7e0dac7a4273"
server_key_iv: "95adb57a8fbfe199cade22d0505b86a5"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.681831 6085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:27.684544 6100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:27.685961 6085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:20:27.686092 6085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "3bfdee97942f4ae2a17c96ace810b662"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "29884a91dcfb8e1bf8db7e0dac7a4273"
server_key_iv: "95adb57a8fbfe199cade22d0505b86a5"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.686200 6085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:27.713172 6085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:27.716512 6085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:27.716735 6085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:27.718230 6085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:27.718305 6085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:27.718377 6085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:27.718451 6085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:27.729418 6085 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:36935
I20260929 16:20:27.729458 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:36935 every 8 connection(s)
I20260929 16:20:27.730579 6085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:27.731938 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.719273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:39427 (local address 127.4.200.126:45693)
0929 16:20:27.719438 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:27.719445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:27.720249 (+ 804us) server_negotiation.cc:407] Connection header received
0929 16:20:27.721391 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:27.721395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:27.721453 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:27.721539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:27.722750 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.723410 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.724610 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.724858 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.724974 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.727989 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:27.727995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:27.727999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:27.728032 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:27.729764 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.730226 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.730233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.730234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.730321 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.730760 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.730763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.730765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.730960 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:27.731085 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:27.731627 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 16:20:27.731755 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:20:27.732677 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.719644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.65:39427)
0929 16:20:27.720090 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:27.720138 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:27.721126 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:27.721725 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:27.721732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:27.722047 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:27.722605 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.722619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.723539 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.723542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.724492 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.724501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.725021 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.725025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.725188 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.725829 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:27.725853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:27.727823 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:27.729868 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.729872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.729883 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.730121 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.730462 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.730465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.730469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.730652 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.731080 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:27.731086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:27.731372 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 16:20:27.731584 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 16:20:27.731882 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":137,"threads_started":1}
I20260929 16:20:27.733605 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6085
I20260929 16:20:27.733716 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:27.734017 4897 external_mini_cluster.cc:1428] Setting key 03a260bbf6d1a431d2f1542786506859
I20260929 16:20:27.734045 6216 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45693
I20260929 16:20:27.734158 6216 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:27.734722 6216 heartbeater.cc:511] Master 127.4.200.126:45693 requested a full tablet report, sending...
I20260929 16:20:27.736333 6012 ts_manager.cc:194] Registered new tserver with Master: 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935)
I20260929 16:20:27.737851 6012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:39427
I20260929 16:20:27.744213 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.736216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.65:52389)
0929 16:20:27.736502 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:27.736518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:27.736603 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:27.737813 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:27.737816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:27.738004 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:27.738097 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.738103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.738916 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.738920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.739475 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.739482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.739930 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.739934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.740028 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.740541 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:27.740552 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:27.740874 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:27.742819 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.742821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.742823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.743018 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.743309 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.743311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.743312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.743385 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.743811 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:27.743813 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:27.743888 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 16:20:27.743966 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:20:27.744074 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260929 16:20:27.744520 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.737507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:52389 (local address 127.4.200.126:45693)
0929 16:20:27.737619 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:27.737623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:27.737636 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:20:27.737672 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:27.737674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:27.737705 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:27.737798 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:27.738209 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.738792 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.739591 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.739819 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.739929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.741018 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:27.741024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:27.741030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:27.741060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:27.742716 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.743116 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.743120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.743122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.743215 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.743484 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.743490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.743492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.743719 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:27.743830 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:27.744068 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:20:27.744175 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:27.761902 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45693
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:27.870859 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:27.871140 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:27.871215 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:27.875283 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:27.875366 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:27.875434 6225 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:27.875517 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:27.880867 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:27.881940 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:27.883105 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:27.893021 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:27.893008 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:27.893039 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:27.893342 6225 server_base.cc:1034] running on GCE node
I20260929 16:20:27.893841 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:27.894606 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:27.895813 6225 hybrid_clock.cc:648] HybridClock initialized: now 1790698827895782 us; error 42 us; skew 500 ppm
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:27.898684 6225 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:27.899612 6225 webserver.cc:460] Webserver started at http://127.4.200.66:44891/ using document root <none> and password file <none>
I20260929 16:20:27.900175 6225 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:27.900260 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:27.900488 6225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:27.902599 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "c3b655622ac84399bfb7990e5dc6afeb"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "f0fb60670bd0c795d6eb2378ae0b5369"
server_key_iv: "3144c77d195135453aacd72d2590928e"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.903167 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "c3b655622ac84399bfb7990e5dc6afeb"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "f0fb60670bd0c795d6eb2378ae0b5369"
server_key_iv: "3144c77d195135453aacd72d2590928e"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.906895 6225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:20:27.909364 6240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:27.910708 6225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:27.910854 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "c3b655622ac84399bfb7990e5dc6afeb"
format_stamp: "Formatted at 2026-09-29 16:20:27 on dist-test-slave-s9ct"
server_key: "f0fb60670bd0c795d6eb2378ae0b5369"
server_key_iv: "3144c77d195135453aacd72d2590928e"
server_key_version: "encryptionkey@0"
I20260929 16:20:27.910969 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:27.933545 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:27.936866 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:27.937021 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:27.938356 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:27.938448 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:27.938515 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:27.938567 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:27 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:27.948900 6225 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:45189
I20260929 16:20:27.948971 6355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:45189 every 8 connection(s)
I20260929 16:20:27.950141 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:27.952739 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.939251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:45539 (local address 127.4.200.126:45693)
0929 16:20:27.939396 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:27.939401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:27.940284 (+ 883us) server_negotiation.cc:407] Connection header received
0929 16:20:27.941462 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:27.941466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:27.941532 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:27.941638 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:27.943457 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.944358 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.945725 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.945955 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.946071 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.949034 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:27.949037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:27.949040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:27.949069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:27.950661 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.951244 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.951248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.951250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.951336 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.951679 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.951682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.951684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.951877 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:27.951983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:27.952456 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:20:27.952576 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:27.953508 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.939678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.66:45539)
0929 16:20:27.940120 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:27.940168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:27.941207 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:27.941974 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:27.941982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:27.942513 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:27.943265 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.943287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.944641 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.944645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.945595 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.945604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.946118 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.946123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.946308 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.947027 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:27.947051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:27.948819 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:27.950788 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.950795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.950811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.951137 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.951431 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.951436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.951438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.951576 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.951980 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:27.951986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:27.952224 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 16:20:27.952416 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 16:20:27.952724 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":132,"threads_started":1}
I20260929 16:20:27.954952 6356 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45693
I20260929 16:20:27.955121 6356 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:27.955636 6356 heartbeater.cc:511] Master 127.4.200.126:45693 requested a full tablet report, sending...
I20260929 16:20:27.956761 6013 ts_manager.cc:194] Registered new tserver with Master: c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189)
I20260929 16:20:27.957545 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:45539
I20260929 16:20:27.958069 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6225
I20260929 16:20:27.958204 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:27.958596 4897 external_mini_cluster.cc:1428] Setting key dad14a4d21faedbffcc1095284217943
I20260929 16:20:27.965492 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.957170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.66:41941)
0929 16:20:27.957473 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:27.957489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:27.957594 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:27.958026 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:27.958030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:27.958278 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:27.958442 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.958452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.959435 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.959438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.960210 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:27.960220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.960833 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.960835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:27.960910 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.961443 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:27.961454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:27.961899 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:27.963925 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.963927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.963929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.964135 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.964432 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:27.964434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:27.964435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:27.964507 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:27.965038 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:27.965040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:27.965112 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 16:20:27.965194 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:20:27.965298 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
I20260929 16:20:27.965621 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:27.957235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:41941 (local address 127.4.200.126:45693)
0929 16:20:27.957381 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:27.957386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:27.957565 (+ 179us) server_negotiation.cc:407] Connection header received
0929 16:20:27.957726 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:27.957729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:27.957792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:27.957887 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:27.958588 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.959311 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.960336 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:27.960712 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:27.960839 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:27.962040 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:27.962043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:27.962045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:27.962066 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:27.963815 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.964232 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.964238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.964243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.964337 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:27.964601 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:27.964605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:27.964607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:27.964946 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:27.965052 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:27.965282 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:20:27.965418 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:27.987597 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45693
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:28.100016 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:28.100353 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:28.100435 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:28.104609 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:28.104719 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:28.104792 6365 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:28.104879 6365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:28.110370 6365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:45693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:28.111552 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:28.112622 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:28.122659 6365 server_base.cc:1034] running on GCE node
W20260929 16:20:28.122627 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:28.122640 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:28.122922 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:28.123330 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:28.123967 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:28.125137 6365 hybrid_clock.cc:648] HybridClock initialized: now 1790698828125104 us; error 49 us; skew 500 ppm
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:28.128032 6365 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:28.128902 6365 webserver.cc:460] Webserver started at http://127.4.200.67:37741/ using document root <none> and password file <none>
I20260929 16:20:28.129472 6365 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:28.129563 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:28.129801 6365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:28.131904 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "70c20fa67e0743fba6c58ed44ba26e71"
format_stamp: "Formatted at 2026-09-29 16:20:28 on dist-test-slave-s9ct"
server_key: "0fa753d77e2d08cb227ef5f82a4dd387"
server_key_iv: "c6e54738cb4c9998d0f89f431ca38167"
server_key_version: "encryptionkey@0"
I20260929 16:20:28.132471 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "70c20fa67e0743fba6c58ed44ba26e71"
format_stamp: "Formatted at 2026-09-29 16:20:28 on dist-test-slave-s9ct"
server_key: "0fa753d77e2d08cb227ef5f82a4dd387"
server_key_iv: "c6e54738cb4c9998d0f89f431ca38167"
server_key_version: "encryptionkey@0"
I20260929 16:20:28.136539 6365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:20:28.139001 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:28.140247 6365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:28.140343 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "70c20fa67e0743fba6c58ed44ba26e71"
format_stamp: "Formatted at 2026-09-29 16:20:28 on dist-test-slave-s9ct"
server_key: "0fa753d77e2d08cb227ef5f82a4dd387"
server_key_iv: "c6e54738cb4c9998d0f89f431ca38167"
server_key_version: "encryptionkey@0"
I20260929 16:20:28.140476 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:28.157634 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:28.160948 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:28.161154 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:28.162627 6365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:28.162698 6365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:28.162760 6365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:28.162796 6365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:28.173693 6365 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:37477
I20260929 16:20:28.173732 6495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:37477 every 8 connection(s)
I20260929 16:20:28.174806 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:28.176012 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.163285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47153 (local address 127.4.200.126:45693)
0929 16:20:28.163443 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.163448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.165031 (+ 1583us) server_negotiation.cc:407] Connection header received
0929 16:20:28.166222 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.166226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.166285 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.166450 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.167723 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.168235 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.169365 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.169605 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.169718 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.172082 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.172084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.172086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.172107 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.173798 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.174344 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.174347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.174348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.174494 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.174906 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.174910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.174912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.175081 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.175192 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.175781 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.175895 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:20:28.176824 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.164352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.67:47153)
0929 16:20:28.164887 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.164928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.166011 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.166619 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.166625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.167101 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:28.167561 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.167574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.168359 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.168363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.169232 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.169241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.169758 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.169762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.169966 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.170572 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:28.170586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:28.171923 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:28.173926 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.173931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.173943 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.174245 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.174588 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.174591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.174593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.174765 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.175208 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:28.175216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:28.175506 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.175722 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.175985 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260929 16:20:28.178045 6496 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45693
I20260929 16:20:28.178187 6496 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:28.178707 6496 heartbeater.cc:511] Master 127.4.200.126:45693 requested a full tablet report, sending...
I20260929 16:20:28.179715 6012 ts_manager.cc:194] Registered new tserver with Master: 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67:37477)
I20260929 16:20:28.180441 6012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:47153
I20260929 16:20:28.184211 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6365
I20260929 16:20:28.184397 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:28.184713 4897 external_mini_cluster.cc:1428] Setting key 258d79fd540722e10854dfd20067f9ad
I20260929 16:20:28.186877 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:20:28.187253 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.180203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:33683 (local address 127.4.200.126:45693)
0929 16:20:28.180312 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.180316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.180439 (+ 123us) server_negotiation.cc:407] Connection header received
0929 16:20:28.180551 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.180554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.180598 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.180689 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.181107 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.181582 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.182580 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.182768 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.182839 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.183874 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.183877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.183878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.183898 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.185584 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.186057 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.186061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.186062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.186147 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.186457 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.186460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.186462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.186646 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.186737 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.187022 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.187123 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:20:28.187268 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.180079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.4.200.67:33683)
0929 16:20:28.180355 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.180369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.180466 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.180712 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.180717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.180872 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:28.180991 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.181000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.181705 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.181708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.182487 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.182496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.182859 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.182862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.182935 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.183450 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:28.183460 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:28.183747 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:28.185677 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.185680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.185681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.185963 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.186233 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.186236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.186238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.186330 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.186746 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:28.186749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:28.186838 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.186965 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.187095 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:28.212810 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.205007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.4.200.126:45693)
0929 16:20:28.205132 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.205136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.205148 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:28.205186 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.205188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.205223 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.205291 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.205941 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.206508 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.207274 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.207471 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.207542 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.209706 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.209709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.209711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.209733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.211171 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.211601 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.211606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.211608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.211693 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.211971 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.211975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.211976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.212146 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.212284 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.212528 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.212642 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:28.215677 6012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:28.218641 6012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:28.232771 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.227637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.4.200.65:36935)
0929 16:20:28.228066 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.228083 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.228106 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:20:28.228182 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.228188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.228373 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.228540 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.228923 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.230054 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.231515 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.232055 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.232179 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.232348 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.232518 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.232623 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":85,"threads_started":1}
I20260929 16:20:28.232761 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.226747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:45189 (local address 127.0.0.1:55654)
0929 16:20:28.227157 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.227211 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.227340 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.228191 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.228197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.228219 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.228379 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.228386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.229813 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.229819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.231193 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.231203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.232263 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.232267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.232377 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.232401 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.232451 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.232588 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":26,"thread_start_us":109,"threads_started":1}
I20260929 16:20:28.233143 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.226922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.4.200.66:45189)
0929 16:20:28.227655 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.227668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.227693 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:20:28.227776 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.227781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.228010 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.228541 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.228672 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.229706 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.231317 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.232139 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.232698 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.232857 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.232908 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.233005 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":106,"threads_started":1}
I20260929 16:20:28.233210 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.227156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:36935 (local address 127.0.0.1:53784)
0929 16:20:28.227425 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.227442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.227536 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.228679 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.228682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.228701 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.228787 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.228792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.230172 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.230176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.231406 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.231414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.232180 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.232184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.232299 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.232339 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.232484 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.232612 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":131,"thread_start_us":103,"threads_started":1}
I20260929 16:20:28.233660 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.227554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37477 (local address 127.0.0.1:35652)
0929 16:20:28.228992 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.229005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.229092 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.229523 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.229525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.229542 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.229617 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.229622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.230832 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.230838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.232058 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.232069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.233257 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.233263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.233364 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.233388 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.233476 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.233544 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260929 16:20:28.234105 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.228007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.4.200.67:37477)
0929 16:20:28.228513 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.228526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.229119 (+ 593us) server_negotiation.cc:407] Connection header received
0929 16:20:28.229199 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.229204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.229378 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.229528 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.229731 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.230716 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.232204 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.232952 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.233046 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.233186 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.233508 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.233596 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":130,"threads_started":1}
I20260929 16:20:28.236479 6290 tablet_service.cc:1467] Processing CreateTablet for tablet ecd567bad4474dcba4bbf02be5355173 (DEFAULT_TABLE table=test-table [id=339df30c8d15427da422c10f183963dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.236471 6430 tablet_service.cc:1467] Processing CreateTablet for tablet ecd567bad4474dcba4bbf02be5355173 (DEFAULT_TABLE table=test-table [id=339df30c8d15427da422c10f183963dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.236464 6148 tablet_service.cc:1467] Processing CreateTablet for tablet ecd567bad4474dcba4bbf02be5355173 (DEFAULT_TABLE table=test-table [id=339df30c8d15427da422c10f183963dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.237478 6290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd567bad4474dcba4bbf02be5355173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.237509 6430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd567bad4474dcba4bbf02be5355173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.237506 6148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd567bad4474dcba4bbf02be5355173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.244392 6519 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Bootstrap starting.
I20260929 16:20:28.245199 6520 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Bootstrap starting.
I20260929 16:20:28.245716 6521 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Bootstrap starting.
I20260929 16:20:28.246774 6519 tablet_bootstrap.cc:654] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.247572 6520 tablet_bootstrap.cc:654] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.247588 6521 tablet_bootstrap.cc:654] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.247723 6519 log.cc:824] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:28.248456 6520 log.cc:824] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:28.248814 6521 log.cc:824] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:28.249668 6519 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: No bootstrap required, opened a new log
I20260929 16:20:28.249861 6519 ts_tablet_manager.cc:1397] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:20:28.250186 6520 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: No bootstrap required, opened a new log
I20260929 16:20:28.250376 6520 ts_tablet_manager.cc:1397] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:20:28.251927 6521 tablet_bootstrap.cc:492] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: No bootstrap required, opened a new log
I20260929 16:20:28.252104 6521 ts_tablet_manager.cc:1397] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:20:28.253245 6520 raft_consensus.cc:348] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.253453 6520 raft_consensus.cc:374] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.253486 6520 raft_consensus.cc:729] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c20fa67e0743fba6c58ed44ba26e71, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.253700 6519 raft_consensus.cc:348] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.253937 6519 raft_consensus.cc:374] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.253994 6519 raft_consensus.cc:729] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b655622ac84399bfb7990e5dc6afeb, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.253906 6520 consensus_queue.cc:260] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.254500 6519 consensus_queue.cc:260] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.254964 6520 ts_tablet_manager.cc:1428] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260929 16:20:28.255450 6496 heartbeater.cc:503] Master 127.4.200.126:45693 was elected leader, sending a full tablet report...
I20260929 16:20:28.255522 6356 heartbeater.cc:503] Master 127.4.200.126:45693 was elected leader, sending a full tablet report...
I20260929 16:20:28.255550 6519 ts_tablet_manager.cc:1428] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:20:28.255872 6521 raft_consensus.cc:348] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.256106 6521 raft_consensus.cc:374] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.256166 6521 raft_consensus.cc:729] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfdee97942f4ae2a17c96ace810b662, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.256587 6521 consensus_queue.cc:260] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.257409 6521 ts_tablet_manager.cc:1428] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:20:28.257747 6216 heartbeater.cc:503] Master 127.4.200.126:45693 was elected leader, sending a full tablet report...
I20260929 16:20:28.420955 6526 raft_consensus.cc:482] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:28.421139 6526 raft_consensus.cc:504] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.422243 6526 leader_election.cc:288] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67:37477), 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935)
W20260929 16:20:28.426034 6497 tablet.cc:2369] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:28.426172 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.422681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:60873 (local address 127.4.200.67:37477)
0929 16:20:28.422819 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.422824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.422840 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:20:28.422912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.422917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.422976 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.423052 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.423358 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.423929 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.424996 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.425573 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.425708 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.425762 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.425985 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.426059 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:20:28.426167 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.422459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37477 (local address 127.4.200.66:60873)
0929 16:20:28.422563 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.422580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.422671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.423079 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.423082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.423101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.423244 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.423251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.424046 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.424049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.424871 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.424880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.425734 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.425738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.425846 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.425872 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.425961 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.426033 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 16:20:28.426975 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.422533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:36935 (local address 127.4.200.66:39233)
0929 16:20:28.422857 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.422871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.422982 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.423236 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.423239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.423252 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.423394 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.423402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.424157 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.424160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.425591 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.425602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.426631 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.426634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.426738 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.426757 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.426826 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.426874 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":70,"threads_started":1}
I20260929 16:20:28.426972 6450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd567bad4474dcba4bbf02be5355173" candidate_uuid: "c3b655622ac84399bfb7990e5dc6afeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c20fa67e0743fba6c58ed44ba26e71" is_pre_election: true
I20260929 16:20:28.427052 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.422658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:39233 (local address 127.4.200.65:36935)
0929 16:20:28.422841 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.422844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.422952 (+ 108us) server_negotiation.cc:407] Connection header received
0929 16:20:28.423089 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.423092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.423137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.423223 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.423515 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.424055 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.425720 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.426350 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.426508 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.426560 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.426860 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.426932 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 16:20:28.427271 6450 raft_consensus.cc:2454] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b655622ac84399bfb7990e5dc6afeb in term 0.
I20260929 16:20:28.427693 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd567bad4474dcba4bbf02be5355173" candidate_uuid: "c3b655622ac84399bfb7990e5dc6afeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662" is_pre_election: true
I20260929 16:20:28.427767 6243 leader_election.cc:302] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70c20fa67e0743fba6c58ed44ba26e71, c3b655622ac84399bfb7990e5dc6afeb; no voters:
I20260929 16:20:28.427983 6170 raft_consensus.cc:2454] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b655622ac84399bfb7990e5dc6afeb in term 0.
I20260929 16:20:28.428068 6526 raft_consensus.cc:2781] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:28.428159 6526 raft_consensus.cc:482] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:28.428210 6526 raft_consensus.cc:3037] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.429241 6526 raft_consensus.cc:504] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.429579 6526 leader_election.cc:288] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [CANDIDATE]: Term 1 election: Requested vote from peers 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67:37477), 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935)
I20260929 16:20:28.429885 6450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd567bad4474dcba4bbf02be5355173" candidate_uuid: "c3b655622ac84399bfb7990e5dc6afeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c20fa67e0743fba6c58ed44ba26e71"
I20260929 16:20:28.429900 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd567bad4474dcba4bbf02be5355173" candidate_uuid: "c3b655622ac84399bfb7990e5dc6afeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662"
I20260929 16:20:28.430023 6450 raft_consensus.cc:3037] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.430032 6170 raft_consensus.cc:3037] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.431106 6170 raft_consensus.cc:2454] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b655622ac84399bfb7990e5dc6afeb in term 1.
I20260929 16:20:28.431433 6241 leader_election.cc:302] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfdee97942f4ae2a17c96ace810b662, c3b655622ac84399bfb7990e5dc6afeb; no voters:
I20260929 16:20:28.431609 6526 raft_consensus.cc:2781] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:28.431653 6450 raft_consensus.cc:2454] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b655622ac84399bfb7990e5dc6afeb in term 1.
I20260929 16:20:28.431823 6526 raft_consensus.cc:686] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 1 LEADER]: Becoming Leader. State: Replica: c3b655622ac84399bfb7990e5dc6afeb, State: Running, Role: LEADER
I20260929 16:20:28.432425 6526 consensus_queue.cc:237] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } }
I20260929 16:20:28.435668 6013 catalog_manager.cc:5495] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66). New cstate: current_term: 1 leader_uuid: "c3b655622ac84399bfb7990e5dc6afeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } health_report { overall_health: UNKNOWN } } }
W20260929 16:20:28.451485 6357 tablet.cc:2369] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:28.479996 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.475983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.4.200.66:45189)
0929 16:20:28.476124 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.476129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.476145 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:20:28.476356 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.476359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.476432 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.476533 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:28.476844 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.477382 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.478358 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.478659 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.478773 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.479132 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.479567 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.479701 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.479784 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.479843 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:20:28.484041 6217 tablet.cc:2369] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:28.486268 6450 raft_consensus.cc:1261] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Refusing update from remote peer c3b655622ac84399bfb7990e5dc6afeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.486267 6170 raft_consensus.cc:1261] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Refusing update from remote peer c3b655622ac84399bfb7990e5dc6afeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.486974 6526 consensus_queue.cc:1035] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.487206 6529 consensus_queue.cc:1035] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.493770 6532 mvcc.cc:205] Tried to move back new op lower bound from 7334702401471549440 to 7334702401262022656. Current Snapshot: MvccSnapshot[applied={T|T < 7334702401471549440}]
I20260929 16:20:28.495085 6534 mvcc.cc:205] Tried to move back new op lower bound from 7334702401471549440 to 7334702401262022656. Current Snapshot: MvccSnapshot[applied={T|T < 7334702401471549440}]
I20260929 16:20:28.496785 6533 mvcc.cc:205] Tried to move back new op lower bound from 7334702401471549440 to 7334702401262022656. Current Snapshot: MvccSnapshot[applied={T|T < 7334702401471549440}]
I20260929 16:20:28.509975 6013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:28.510166 6013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:28.512951 6013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Sending DeleteTablet for 3 replicas of tablet ecd567bad4474dcba4bbf02be5355173
I20260929 16:20:28.513859 6148 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd567bad4474dcba4bbf02be5355173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53784
I20260929 16:20:28.513870 6430 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd567bad4474dcba4bbf02be5355173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 16:20:28.513845 6290 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd567bad4474dcba4bbf02be5355173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:55654
I20260929 16:20:28.514485 6542 tablet_replica.cc:330] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: stopping tablet replica
I20260929 16:20:28.514910 6542 raft_consensus.cc:2229] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:28.515264 6542 raft_consensus.cc:2258] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.515575 6013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:20:28.515690 6543 tablet_replica.cc:330] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: stopping tablet replica
I20260929 16:20:28.515974 6543 raft_consensus.cc:2229] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 1 LEADER]: Raft consensus shutting down.
W20260929 16:20:28.516033 6013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:28.516388 6543 raft_consensus.cc:2258] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.518252 6542 ts_tablet_manager.cc:1905] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.520908 6148 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9e05026624b32bf2202a56645415a (DEFAULT_TABLE table=test-table [id=82a4e99218c44a5ebfdf9b6df137206f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.521092 6290 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9e05026624b32bf2202a56645415a (DEFAULT_TABLE table=test-table [id=82a4e99218c44a5ebfdf9b6df137206f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.521188 6148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9e05026624b32bf2202a56645415a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.521147 6430 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9e05026624b32bf2202a56645415a (DEFAULT_TABLE table=test-table [id=82a4e99218c44a5ebfdf9b6df137206f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:28.521332 6290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9e05026624b32bf2202a56645415a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.521390 6430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c9e05026624b32bf2202a56645415a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.523463 6521 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Bootstrap starting.
I20260929 16:20:28.523677 6519 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Bootstrap starting.
I20260929 16:20:28.523983 6520 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Bootstrap starting.
I20260929 16:20:28.524628 6519 tablet_bootstrap.cc:654] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.524886 6521 tablet_bootstrap.cc:654] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.525439 6520 tablet_bootstrap.cc:654] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.526647 6543 ts_tablet_manager.cc:1905] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.527511 6521 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: No bootstrap required, opened a new log
I20260929 16:20:28.527594 6519 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: No bootstrap required, opened a new log
I20260929 16:20:28.527621 6521 ts_tablet_manager.cc:1397] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:20:28.527675 6519 ts_tablet_manager.cc:1397] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:20:28.528036 6521 raft_consensus.cc:348] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.528159 6521 raft_consensus.cc:374] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.528110 6519 raft_consensus.cc:348] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.528200 6521 raft_consensus.cc:729] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfdee97942f4ae2a17c96ace810b662, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.528208 6519 raft_consensus.cc:374] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.528240 6519 raft_consensus.cc:729] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b655622ac84399bfb7990e5dc6afeb, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.528328 6521 consensus_queue.cc:260] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.528355 6519 consensus_queue.cc:260] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.528596 6519 ts_tablet_manager.cc:1428] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:28.528606 6521 ts_tablet_manager.cc:1428] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:28.529696 6520 tablet_bootstrap.cc:492] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: No bootstrap required, opened a new log
I20260929 16:20:28.529783 6520 ts_tablet_manager.cc:1397] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 16:20:28.530200 6520 raft_consensus.cc:348] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.530315 6520 raft_consensus.cc:374] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.530357 6520 raft_consensus.cc:729] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c20fa67e0743fba6c58ed44ba26e71, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.530505 6520 consensus_queue.cc:260] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.530752 6520 ts_tablet_manager.cc:1428] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:28.531925 6542 ts_tablet_manager.cc:1918] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:28.532027 6542 log.cc:1196] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/ecd567bad4474dcba4bbf02be5355173
I20260929 16:20:28.532337 6542 ts_tablet_manager.cc:1939] T ecd567bad4474dcba4bbf02be5355173 P 3bfdee97942f4ae2a17c96ace810b662: Deleting consensus metadata
I20260929 16:20:28.533772 5997 catalog_manager.cc:4841] TS 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935): tablet ecd567bad4474dcba4bbf02be5355173 (table test-table [id=339df30c8d15427da422c10f183963dd]) successfully deleted
I20260929 16:20:28.534570 6543 ts_tablet_manager.cc:1918] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:28.534586 6541 tablet_replica.cc:330] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: stopping tablet replica
I20260929 16:20:28.534669 6543 log.cc:1196] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/ecd567bad4474dcba4bbf02be5355173
I20260929 16:20:28.534920 6541 raft_consensus.cc:2229] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:28.534997 6543 ts_tablet_manager.cc:1939] T ecd567bad4474dcba4bbf02be5355173 P c3b655622ac84399bfb7990e5dc6afeb: Deleting consensus metadata
I20260929 16:20:28.535270 6541 raft_consensus.cc:2258] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.536202 5999 catalog_manager.cc:4841] TS c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189): tablet ecd567bad4474dcba4bbf02be5355173 (table test-table [id=339df30c8d15427da422c10f183963dd]) successfully deleted
I20260929 16:20:28.536897 6541 ts_tablet_manager.cc:1905] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.538939 6541 ts_tablet_manager.cc:1918] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:28.539060 6541 log.cc:1196] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/ecd567bad4474dcba4bbf02be5355173
I20260929 16:20:28.539330 6541 ts_tablet_manager.cc:1939] T ecd567bad4474dcba4bbf02be5355173 P 70c20fa67e0743fba6c58ed44ba26e71: Deleting consensus metadata
I20260929 16:20:28.540455 5999 catalog_manager.cc:4841] TS 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67:37477): tablet ecd567bad4474dcba4bbf02be5355173 (table test-table [id=339df30c8d15427da422c10f183963dd]) successfully deleted
I20260929 16:20:28.656481 6525 raft_consensus.cc:482] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:28.656634 6525 raft_consensus.cc:504] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.657787 6525 leader_election.cc:288] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935), c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189)
I20260929 16:20:28.662156 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.658250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:60639 (local address 127.4.200.66:45189)
0929 16:20:28.658412 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.658417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.658449 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:20:28.658509 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.658515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.658569 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.658644 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.659024 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.659627 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.661030 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.661582 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.661715 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.661774 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.661948 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.662029 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:20:28.662154 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.658044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:45189 (local address 127.4.200.67:60639)
0929 16:20:28.658166 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.658184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.658283 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.658768 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.658772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.658786 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.658918 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.658924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.659730 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.659735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.660835 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.660846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.661710 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.661713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.661817 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.661842 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.661922 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.662002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 16:20:28.662575 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.658236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:45049 (local address 127.4.200.65:36935)
0929 16:20:28.658357 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.658362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.658778 (+ 416us) server_negotiation.cc:407] Connection header received
0929 16:20:28.658865 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.658869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.658934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.659044 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:28.659310 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.660090 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.661361 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.661838 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.661958 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.662003 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.662334 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.662466 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:20:28.662961 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.658047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:36935 (local address 127.4.200.67:45049)
0929 16:20:28.658659 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.658672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.658767 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.659059 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.659062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.659079 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:28.659186 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.659193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.660236 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.660239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.661238 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.661245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.662074 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.662077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.662180 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.662209 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.662710 (+ 501us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.662799 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":326,"thread_start_us":81,"threads_started":1}
I20260929 16:20:28.663239 6310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9e05026624b32bf2202a56645415a" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b655622ac84399bfb7990e5dc6afeb" is_pre_election: true
I20260929 16:20:28.663444 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9e05026624b32bf2202a56645415a" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662" is_pre_election: true
I20260929 16:20:28.663506 6310 raft_consensus.cc:2454] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 0.
I20260929 16:20:28.663586 6170 raft_consensus.cc:2454] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 0.
I20260929 16:20:28.663964 6381 leader_election.cc:302] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bfdee97942f4ae2a17c96ace810b662, 70c20fa67e0743fba6c58ed44ba26e71, c3b655622ac84399bfb7990e5dc6afeb; no voters:
I20260929 16:20:28.664276 6525 raft_consensus.cc:2781] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:28.664340 6525 raft_consensus.cc:482] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:28.664377 6525 raft_consensus.cc:3037] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.665361 6525 raft_consensus.cc:504] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.665776 6525 leader_election.cc:288] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935), c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189)
I20260929 16:20:28.666149 6310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9e05026624b32bf2202a56645415a" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b655622ac84399bfb7990e5dc6afeb"
I20260929 16:20:28.666148 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9e05026624b32bf2202a56645415a" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662"
I20260929 16:20:28.666283 6310 raft_consensus.cc:3037] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.666316 6170 raft_consensus.cc:3037] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.667239 6310 raft_consensus.cc:2454] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 1.
I20260929 16:20:28.667555 6170 raft_consensus.cc:2454] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 1.
I20260929 16:20:28.667598 6383 leader_election.cc:302] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70c20fa67e0743fba6c58ed44ba26e71, c3b655622ac84399bfb7990e5dc6afeb; no voters:
I20260929 16:20:28.667789 6525 raft_consensus.cc:2781] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:28.667970 6525 raft_consensus.cc:686] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 LEADER]: Becoming Leader. State: Replica: 70c20fa67e0743fba6c58ed44ba26e71, State: Running, Role: LEADER
I20260929 16:20:28.668232 6525 consensus_queue.cc:237] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.670823 6012 catalog_manager.cc:5495] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67). New cstate: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } health_report { overall_health: HEALTHY } } }
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698827, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:28.734011 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.724520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.4.200.126:45693)
0929 16:20:28.724843 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.724862 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.724984 (+ 122us) server_negotiation.cc:407] Connection header received
0929 16:20:28.725159 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.725162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.725209 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.725312 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.726019 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.726750 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.727623 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.727876 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.727994 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.730499 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.730503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.730505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.730528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.731980 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.732530 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.732532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.732534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.732608 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.732921 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.732924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.732925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.733221 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.733503 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.733673 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.733857 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":238,"thread_start_us":144,"threads_started":1}
I20260929 16:20:28.734018 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.724498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45693 (local address 127.0.0.1:57450)
0929 16:20:28.724864 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.724892 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.725053 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.725329 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.725332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.725779 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:28.725891 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.725898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.726898 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.726904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.727499 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.727507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.727997 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.728000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.728090 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.728660 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:28.728673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:28.730302 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:28.732098 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.732107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.732113 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.732423 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.732701 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:28.732704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:28.732707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:28.732806 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:28.733349 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:28.733355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:28.733487 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.733614 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.733787 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260929 16:20:28.738390 6012 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:57450:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:20:28.739156 6012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:28.743914 6148 tablet_service.cc:1467] Processing CreateTablet for tablet d85725130e0b464f9b0e43e2c8328f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4ff294ba3bc43ecb4b899c0b0ec5969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:28.744143 6430 tablet_service.cc:1467] Processing CreateTablet for tablet d85725130e0b464f9b0e43e2c8328f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4ff294ba3bc43ecb4b899c0b0ec5969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:28.744323 6148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85725130e0b464f9b0e43e2c8328f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.744164 6290 tablet_service.cc:1467] Processing CreateTablet for tablet d85725130e0b464f9b0e43e2c8328f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4ff294ba3bc43ecb4b899c0b0ec5969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:28.744477 6430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85725130e0b464f9b0e43e2c8328f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.744585 6290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85725130e0b464f9b0e43e2c8328f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:28.747089 6519 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: Bootstrap starting.
I20260929 16:20:28.747193 6521 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: Bootstrap starting.
I20260929 16:20:28.747349 6520 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: Bootstrap starting.
I20260929 16:20:28.747987 6519 tablet_bootstrap.cc:654] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.748051 6521 tablet_bootstrap.cc:654] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.748195 6520 tablet_bootstrap.cc:654] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:28.749393 6521 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: No bootstrap required, opened a new log
I20260929 16:20:28.749462 6521 ts_tablet_manager.cc:1397] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:28.749854 6521 raft_consensus.cc:348] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.749948 6521 raft_consensus.cc:374] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.749969 6521 raft_consensus.cc:729] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfdee97942f4ae2a17c96ace810b662, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.750115 6521 consensus_queue.cc:260] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.750416 6521 ts_tablet_manager.cc:1428] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:28.750341 6527 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.750499 6527 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.751559 6520 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: No bootstrap required, opened a new log
I20260929 16:20:28.751664 6520 ts_tablet_manager.cc:1397] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:20:28.752586 6520 raft_consensus.cc:348] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.752703 6520 raft_consensus.cc:374] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.752741 6520 raft_consensus.cc:729] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c20fa67e0743fba6c58ed44ba26e71, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.752897 6520 consensus_queue.cc:260] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.753124 6525 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.753218 6519 tablet_bootstrap.cc:492] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: No bootstrap required, opened a new log
I20260929 16:20:28.753247 6525 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.753299 6519 ts_tablet_manager.cc:1397] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 16:20:28.753742 6520 ts_tablet_manager.cc:1428] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:28.753710 6519 raft_consensus.cc:348] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.753846 6519 raft_consensus.cc:374] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:28.753896 6519 raft_consensus.cc:729] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b655622ac84399bfb7990e5dc6afeb, State: Initialized, Role: FOLLOWER
I20260929 16:20:28.754024 6519 consensus_queue.cc:260] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.754246 6526 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.754384 6526 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.754315 6519 ts_tablet_manager.cc:1428] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:28.792332 6525 raft_consensus.cc:482] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:28.792498 6525 raft_consensus.cc:504] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.793008 6525 leader_election.cc:288] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935), c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189)
I20260929 16:20:28.793491 6310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85725130e0b464f9b0e43e2c8328f92" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b655622ac84399bfb7990e5dc6afeb" is_pre_election: true
I20260929 16:20:28.793637 6310 raft_consensus.cc:2454] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 0.
I20260929 16:20:28.793515 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85725130e0b464f9b0e43e2c8328f92" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662" is_pre_election: true
I20260929 16:20:28.793838 6170 raft_consensus.cc:2454] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 0.
I20260929 16:20:28.793949 6383 leader_election.cc:302] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70c20fa67e0743fba6c58ed44ba26e71, c3b655622ac84399bfb7990e5dc6afeb; no voters:
I20260929 16:20:28.794119 6525 raft_consensus.cc:2781] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:28.794164 6525 raft_consensus.cc:482] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:28.794200 6525 raft_consensus.cc:3037] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.795727 6525 raft_consensus.cc:504] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.796176 6525 leader_election.cc:288] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935), c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189)
I20260929 16:20:28.796712 6170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85725130e0b464f9b0e43e2c8328f92" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfdee97942f4ae2a17c96ace810b662"
I20260929 16:20:28.796736 6310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85725130e0b464f9b0e43e2c8328f92" candidate_uuid: "70c20fa67e0743fba6c58ed44ba26e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b655622ac84399bfb7990e5dc6afeb"
I20260929 16:20:28.796833 6170 raft_consensus.cc:3037] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.796897 6310 raft_consensus.cc:3037] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:28.797705 6170 raft_consensus.cc:2454] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 1.
I20260929 16:20:28.798018 6381 leader_election.cc:302] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfdee97942f4ae2a17c96ace810b662, 70c20fa67e0743fba6c58ed44ba26e71; no voters:
I20260929 16:20:28.798233 6310 raft_consensus.cc:2454] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c20fa67e0743fba6c58ed44ba26e71 in term 1.
I20260929 16:20:28.798213 6525 raft_consensus.cc:2781] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:28.798346 6525 raft_consensus.cc:686] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 LEADER]: Becoming Leader. State: Replica: 70c20fa67e0743fba6c58ed44ba26e71, State: Running, Role: LEADER
I20260929 16:20:28.798547 6525 consensus_queue.cc:237] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } }
I20260929 16:20:28.799319 6548 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: TxnStatusTablet state changed. Reason: New leader 70c20fa67e0743fba6c58ed44ba26e71. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.799433 6548 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:28.800099 6561 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:20:28.800263 6561 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:20:28.800568 6013 catalog_manager.cc:5495] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67). New cstate: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } health_report { overall_health: HEALTHY } } }
I20260929 16:20:28.825186 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.820118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37477 (local address 127.0.0.1:35660)
0929 16:20:28.820375 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.820389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.820501 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.820845 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.820849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.820856 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:28.820973 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.820982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.822119 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.822124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.823182 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.823193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.823804 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.823808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.823906 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.823983 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.824894 (+ 911us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.824911 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.824981 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.825071 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 16:20:28.825186 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.820223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.4.200.67:37477)
0929 16:20:28.820347 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.820353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.820432 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:20:28.820622 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.820624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.820715 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.820820 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:28.821083 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.821999 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.823289 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.823679 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.823794 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.824130 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.824801 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.824918 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.824990 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.825042 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:20:28.828536 6170 raft_consensus.cc:1261] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Refusing update from remote peer 70c20fa67e0743fba6c58ed44ba26e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.828706 6310 raft_consensus.cc:1261] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Refusing update from remote peer 70c20fa67e0743fba6c58ed44ba26e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.829128 6525 consensus_queue.cc:1035] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.829312 6548 consensus_queue.cc:1035] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.830719 6527 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: TxnStatusTablet state changed. Reason: New leader 70c20fa67e0743fba6c58ed44ba26e71. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.830842 6527 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.831483 6526 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: TxnStatusTablet state changed. Reason: New leader 70c20fa67e0743fba6c58ed44ba26e71. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.831594 6526 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.833268 6525 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.833386 6525 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:28.834053 6548 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.834144 6548 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 70c20fa67e0743fba6c58ed44ba26e71: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:28.834082 6527 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.834182 6527 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P 3bfdee97942f4ae2a17c96ace810b662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.834978 6526 tablet_replica.cc:439] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70c20fa67e0743fba6c58ed44ba26e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 } } peers { permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 } } peers { permanent_uuid: "70c20fa67e0743fba6c58ed44ba26e71" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37477 } } }
I20260929 16:20:28.835103 6526 tablet_replica.cc:442] T d85725130e0b464f9b0e43e2c8328f92 P c3b655622ac84399bfb7990e5dc6afeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:28.844347 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.840547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.4.200.67:37477)
0929 16:20:28.840652 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.840656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.840715 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:20:28.840851 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.840854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.840896 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.840990 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:28.841224 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.841833 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.843024 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.843374 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.843489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.843844 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.843959 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.844042 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.844131 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.844183 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:20:28.855363 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.851602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:35439 (local address 127.4.200.67:37477)
0929 16:20:28.851724 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.851728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.851991 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:20:28.852128 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.852130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.852196 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.852274 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:28.852557 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.853291 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.854140 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.854505 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.854653 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.854932 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.855009 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.855085 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.855162 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.855209 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":37}
I20260929 16:20:28.855379 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.851571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37477 (local address 127.4.200.67:35439)
0929 16:20:28.851893 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:28.851911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:28.852037 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:28.852277 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:28.852280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:28.852286 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:28.852451 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.852457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.853385 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.853390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.854032 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:28.854040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.854612 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.854615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:28.854716 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.854798 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:28.855086 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:28.855092 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:20:28.855156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:20:28.855230 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":27,"thread_start_us":114,"threads_started":1}
I20260929 16:20:28.863225 6170 raft_consensus.cc:1261] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Refusing update from remote peer 70c20fa67e0743fba6c58ed44ba26e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.863601 6310 raft_consensus.cc:1261] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Refusing update from remote peer 70c20fa67e0743fba6c58ed44ba26e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:28.863695 6569 consensus_queue.cc:1035] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfdee97942f4ae2a17c96ace810b662" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.864059 6569 consensus_queue.cc:1035] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b655622ac84399bfb7990e5dc6afeb" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:28.905078 6013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:28.905228 6013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:28.907267 6013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d01a44874654b0895291c86f587a023: Sending DeleteTablet for 3 replicas of tablet 16c9e05026624b32bf2202a56645415a
I20260929 16:20:28.907819 6148 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9e05026624b32bf2202a56645415a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53784
I20260929 16:20:28.908015 6542 tablet_replica.cc:330] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: stopping tablet replica
I20260929 16:20:28.907989 6290 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9e05026624b32bf2202a56645415a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:55654
I20260929 16:20:28.908113 6429 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9e05026624b32bf2202a56645415a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 16:20:28.908143 6542 raft_consensus.cc:2229] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:28.908169 6543 tablet_replica.cc:330] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: stopping tablet replica
I20260929 16:20:28.908264 6541 tablet_replica.cc:330] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: stopping tablet replica
I20260929 16:20:28.908259 6542 raft_consensus.cc:2258] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.908267 6543 raft_consensus.cc:2229] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:28.908362 6541 raft_consensus.cc:2229] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:28.908370 6543 raft_consensus.cc:2258] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.908648 6541 raft_consensus.cc:2258] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:28.908973 6542 ts_tablet_manager.cc:1905] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.909495 6541 ts_tablet_manager.cc:1905] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.911798 6542 ts_tablet_manager.cc:1918] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:28.911878 6542 log.cc:1196] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/16c9e05026624b32bf2202a56645415a
I20260929 16:20:28.912057 6541 ts_tablet_manager.cc:1918] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:28.912132 6541 log.cc:1196] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/16c9e05026624b32bf2202a56645415a
I20260929 16:20:28.912164 6542 ts_tablet_manager.cc:1939] T 16c9e05026624b32bf2202a56645415a P 3bfdee97942f4ae2a17c96ace810b662: Deleting consensus metadata
I20260929 16:20:28.912376 6541 ts_tablet_manager.cc:1939] T 16c9e05026624b32bf2202a56645415a P 70c20fa67e0743fba6c58ed44ba26e71: Deleting consensus metadata
I20260929 16:20:28.912425 6543 ts_tablet_manager.cc:1905] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:28.913188 5997 catalog_manager.cc:4841] TS 3bfdee97942f4ae2a17c96ace810b662 (127.4.200.65:36935): tablet 16c9e05026624b32bf2202a56645415a (table test-table [id=82a4e99218c44a5ebfdf9b6df137206f]) successfully deleted
I20260929 16:20:28.913188 5999 catalog_manager.cc:4841] TS 70c20fa67e0743fba6c58ed44ba26e71 (127.4.200.67:37477): tablet 16c9e05026624b32bf2202a56645415a (table test-table [id=82a4e99218c44a5ebfdf9b6df137206f]) successfully deleted
I20260929 16:20:28.915330 6543 ts_tablet_manager.cc:1918] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:28.915409 6543 log.cc:1196] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/16c9e05026624b32bf2202a56645415a
I20260929 16:20:28.915679 6543 ts_tablet_manager.cc:1939] T 16c9e05026624b32bf2202a56645415a P c3b655622ac84399bfb7990e5dc6afeb: Deleting consensus metadata
I20260929 16:20:28.916307 5999 catalog_manager.cc:4841] TS c3b655622ac84399bfb7990e5dc6afeb (127.4.200.66:45189): tablet 16c9e05026624b32bf2202a56645415a (table test-table [id=82a4e99218c44a5ebfdf9b6df137206f]) successfully deleted
Sep 29 16:20:28 dist-test-slave-s9ct krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:28.924499 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.913556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.4.200.65:36935)
0929 16:20:28.913692 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.913697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.913978 (+ 281us) server_negotiation.cc:407] Connection header received
0929 16:20:28.914199 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.914202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.914254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.914338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.915455 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.916221 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.917128 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.917361 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.917472 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.920905 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.920917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.920934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.920966 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.922719 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.923147 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.923155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.923162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.923244 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.923524 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.923531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.923534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.923783 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.924005 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.924209 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.924330 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 16:20:28.924925 6208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53792
I20260929 16:20:28.934619 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:28.926691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.4.200.126:45693)
0929 16:20:28.926821 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:28.926827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:28.926961 (+ 134us) server_negotiation.cc:407] Connection header received
0929 16:20:28.927091 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:28.927094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:28.927144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:28.927222 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:28.927992 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.928761 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.929521 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:28.929868 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:28.929973 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:28.931235 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:28.931239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:28.931242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:28.931269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:28.932838 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.933230 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.933234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.933236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.933355 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:28.933630 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:28.933633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:28.933635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:28.933867 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:28.934003 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:28.934271 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:20:28.934457 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 16:20:28.935017 6013 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57464
I20260929 16:20:28.936414 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6085
I20260929 16:20:28.945048 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6225
I20260929 16:20:28.953979 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6365
I20260929 16:20:28.961254 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 5981
2026-09-29T16:20:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2848 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:29 dist-test-slave-s9ct krb5kdc[6579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:29 dist-test-slave-s9ct krb5kdc[6579](info): set up 2 sockets
Sep 29 16:20:29 dist-test-slave-s9ct krb5kdc[6579](info): commencing operation
krb5kdc: starting...
W20260929 16:20:30.002272 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:30.130040 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36361
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:30.241879 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:30.242146 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:30.242187 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:30.246304 6592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:30.246367 6592 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:20:30.246384 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:30.246460 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:30.246479 6592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:30.246500 6592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:30.251832 6592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:30.252843 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:30.253854 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:30.262709 6598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.262756 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.262858 6592 server_base.cc:1034] running on GCE node
W20260929 16:20:30.262742 6597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.263604 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:30.264808 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:30.266072 6592 hybrid_clock.cc:648] HybridClock initialized: now 1790698830266055 us; error 32 us; skew 500 ppm
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:30.275133 6592 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:30.276090 6592 webserver.cc:460] Webserver started at http://127.4.200.126:37809/ using document root <none> and password file <none>
I20260929 16:20:30.276715 6592 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:30.276780 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:30.277029 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:30.279222 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "accc93c2d5fe47de991687bee8f2e36d"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "6cf6c3ec0b0d4f4ce1e91a049715ffdc"
server_key_iv: "e4ec5e3f83fee1128cad15c4c5bca9a8"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.279808 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "accc93c2d5fe47de991687bee8f2e36d"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "6cf6c3ec0b0d4f4ce1e91a049715ffdc"
server_key_iv: "e4ec5e3f83fee1128cad15c4c5bca9a8"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.283631 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 16:20:30.286168 6607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:30.287531 6592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:20:30.287664 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "accc93c2d5fe47de991687bee8f2e36d"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "6cf6c3ec0b0d4f4ce1e91a049715ffdc"
server_key_iv: "e4ec5e3f83fee1128cad15c4c5bca9a8"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.287776 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:30.297533 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:30.300606 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:30.300800 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:30.308712 6592 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36361
I20260929 16:20:30.308729 6669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36361 every 8 connection(s)
I20260929 16:20:30.309685 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:30.312454 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:30.316502 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6592
I20260929 16:20:30.316603 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:30.316861 4897 external_mini_cluster.cc:1428] Setting key 46dce9c621276566cbc3302ebd3fd5f6
I20260929 16:20:30.318320 6670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Bootstrap starting.
I20260929 16:20:30.321127 6670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:30.322005 6670 log.cc:824] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:30.323880 6670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: No bootstrap required, opened a new log
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:30.328063 6670 raft_consensus.cc:348] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } }
I20260929 16:20:30.328310 6670 raft_consensus.cc:374] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:30.328400 6670 raft_consensus.cc:729] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accc93c2d5fe47de991687bee8f2e36d, State: Initialized, Role: FOLLOWER
I20260929 16:20:30.328835 6670 consensus_queue.cc:260] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } }
I20260929 16:20:30.328982 6670 raft_consensus.cc:388] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:30.329051 6670 raft_consensus.cc:482] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:30.329169 6670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:30.330058 6670 raft_consensus.cc:504] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } }
I20260929 16:20:30.330492 6670 leader_election.cc:302] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: accc93c2d5fe47de991687bee8f2e36d; no voters:
I20260929 16:20:30.330883 6670 leader_election.cc:288] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:30.331102 6675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:30.331310 6675 raft_consensus.cc:686] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [term 1 LEADER]: Becoming Leader. State: Replica: accc93c2d5fe47de991687bee8f2e36d, State: Running, Role: LEADER
I20260929 16:20:30.331619 6675 consensus_queue.cc:237] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } }
I20260929 16:20:30.332029 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.318259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.4.200.126:36361)
0929 16:20:30.318745 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:30.318764 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:30.318806 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:20:30.319660 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:30.319688 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:30.320100 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:30.320462 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:30.321118 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.322617 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.323491 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.323964 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.324144 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.326895 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:30.326901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:30.326913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:30.326939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:30.328818 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.329329 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.329335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.329349 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.329467 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.329753 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.329758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.329759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.330164 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:30.330375 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:30.330693 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:20:30.331023 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":173,"threads_started":1}
I20260929 16:20:30.332157 6670 sys_catalog.cc:564] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:30.332814 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader accc93c2d5fe47de991687bee8f2e36d. Latest consensus state: current_term: 1 leader_uuid: "accc93c2d5fe47de991687bee8f2e36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } } }
I20260929 16:20:30.332933 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:30.333276 6676 sys_catalog.cc:455] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "accc93c2d5fe47de991687bee8f2e36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accc93c2d5fe47de991687bee8f2e36d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36361 } } }
I20260929 16:20:30.333371 6676 sys_catalog.cc:458] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d [sys.catalog]: This master's current role is: LEADER
W20260929 16:20:30.337005 6690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:30.337086 6690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:30.337172 6681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:30.337952 6681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:30.344513 6681 catalog_manager.cc:1347] Generated new cluster ID: d8516f66c93649fbbbe9627eb174f74b
I20260929 16:20:30.344589 6681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:30.358412 6681 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:30.359786 6681 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:30.374910 6681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Generated new TSK 0
I20260929 16:20:30.375841 6681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:30.414353 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36361
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:30.528867 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:30.529131 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:30.529201 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:30.533509 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:30.533578 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:30.533653 6696 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:20:30.533695 6696 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:30.533757 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:30.538934 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:30.540014 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:30.541023 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:30.550343 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.550345 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.550496 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.550758 6696 server_base.cc:1034] running on GCE node
I20260929 16:20:30.551147 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:30.551769 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:30.552956 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790698830552914 us; error 51 us; skew 500 ppm
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:30.555940 6696 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:30.556826 6696 webserver.cc:460] Webserver started at http://127.4.200.65:34125/ using document root <none> and password file <none>
I20260929 16:20:30.557390 6696 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:30.557482 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:30.557714 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:30.559760 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "f808264d9be546ae8ce4e9f5526d80c1"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "2378d18d972e97f9829187463258b918"
server_key_iv: "26f9adf72fe9f86ca5a8bf82ee0d557b"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.560316 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "f808264d9be546ae8ce4e9f5526d80c1"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "2378d18d972e97f9829187463258b918"
server_key_iv: "26f9adf72fe9f86ca5a8bf82ee0d557b"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.564126 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:20:30.566536 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:30.567757 6696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:30.567919 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "f808264d9be546ae8ce4e9f5526d80c1"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "2378d18d972e97f9829187463258b918"
server_key_iv: "26f9adf72fe9f86ca5a8bf82ee0d557b"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.568042 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:30.586921 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:30.589743 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:30.589958 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:30.591444 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:30.591516 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:30.591585 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:30.591609 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:30.602595 6696 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:37991
I20260929 16:20:30.603765 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:30.604636 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:37991 every 8 connection(s)
I20260929 16:20:30.605671 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.592402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56471 (local address 127.4.200.126:36361)
0929 16:20:30.592527 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:30.592538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:30.593644 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 16:20:30.594917 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:30.594921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:30.594981 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:30.595084 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:30.596292 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.597051 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.598212 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.598552 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.598670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.601839 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:30.601842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:30.601844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:30.601870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:30.603595 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.604169 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.604175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.604179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.604278 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.604687 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.604690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.604692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.604912 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:30.604995 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:30.605450 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:20:30.605553 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:20:30.606530 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.592999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.65:56471)
0929 16:20:30.593490 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:30.593531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:30.594673 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:30.595258 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:30.595265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:30.595660 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:30.596127 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.596140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.597183 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.597187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.598078 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.598087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.598724 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.598728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.598901 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.599514 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:30.599531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:30.601655 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:30.603718 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.603723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.603734 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.604062 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.604406 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.604410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.604414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.604592 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.605009 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:30.605014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:30.605230 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 16:20:30.605388 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:20:30.605629 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260929 16:20:30.607882 6827 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36361
I20260929 16:20:30.608057 6827 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:30.608567 6827 heartbeater.cc:511] Master 127.4.200.126:36361 requested a full tablet report, sending...
I20260929 16:20:30.610148 6624 ts_manager.cc:194] Registered new tserver with Master: f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991)
I20260929 16:20:30.611088 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6696
I20260929 16:20:30.611210 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:30.611464 4897 external_mini_cluster.cc:1428] Setting key 0952fba7bd04bdd3a8bbad6c18729332
I20260929 16:20:30.611522 6624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:56471
I20260929 16:20:30.617825 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.609987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:38127 (local address 127.4.200.126:36361)
0929 16:20:30.610124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:30.610129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:30.610410 (+ 281us) server_negotiation.cc:407] Connection header received
0929 16:20:30.610542 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:30.610545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:30.610592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:30.610674 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:30.611202 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.611898 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.612907 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.613208 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.613327 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.614541 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:30.614544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:30.614547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:30.614571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:30.616198 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.616693 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.616695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.616696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.616768 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.617085 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.617088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.617089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.617245 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:30.617321 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:30.617614 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:20:30.617707 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:20:30.618049 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.609914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.65:38127)
0929 16:20:30.610300 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:30.610313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:30.610452 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:30.610690 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:30.610693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:30.610920 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:30.611068 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.611078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.612044 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.612050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.612739 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.612754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.613329 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.613332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.613416 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.613954 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:30.613966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:30.614364 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:30.616316 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.616323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.616328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.616594 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.616863 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.616867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.616877 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.616986 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.617344 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:30.617350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:30.617447 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:20:30.617567 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:20:30.617741 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:30.641255 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36361
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:30.752281 6836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:30.752532 6836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:30.752614 6836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:30.757398 6836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:30.757457 6836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:30.757498 6836 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:20:30.757547 6836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:30.757611 6836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:30.763000 6836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:30.764096 6836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:30.765153 6836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:30.774909 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.774919 6842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.775151 6836 server_base.cc:1034] running on GCE node
W20260929 16:20:30.774920 6841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.775557 6836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:30.776233 6836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:30.777393 6836 hybrid_clock.cc:648] HybridClock initialized: now 1790698830777379 us; error 26 us; skew 500 ppm
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:30.780264 6836 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:30.781126 6836 webserver.cc:460] Webserver started at http://127.4.200.66:41253/ using document root <none> and password file <none>
I20260929 16:20:30.781688 6836 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:30.781777 6836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:30.782006 6836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:30.784114 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "ea0f010b77bf34ae91f85a90749212bd"
server_key_iv: "ad0fed37f1c776fdeed5f7745a1f5482"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.784690 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "ea0f010b77bf34ae91f85a90749212bd"
server_key_iv: "ad0fed37f1c776fdeed5f7745a1f5482"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.788600 6836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 16:20:30.791251 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:30.792601 6836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:20:30.792730 6836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
format_stamp: "Formatted at 2026-09-29 16:20:30 on dist-test-slave-s9ct"
server_key: "ea0f010b77bf34ae91f85a90749212bd"
server_key_iv: "ad0fed37f1c776fdeed5f7745a1f5482"
server_key_version: "encryptionkey@0"
I20260929 16:20:30.792841 6836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:30.812896 6836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:30.816138 6836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:30.816378 6836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:30.817943 6836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:30.818012 6836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:30.818089 6836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:30.818137 6836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:30 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:30.829015 6836 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:35607
I20260929 16:20:30.829083 6966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:35607 every 8 connection(s)
I20260929 16:20:30.829939 6836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:30.832727 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.818770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:53691 (local address 127.4.200.126:36361)
0929 16:20:30.818923 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:30.818928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:30.819899 (+ 971us) server_negotiation.cc:407] Connection header received
0929 16:20:30.821233 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:30.821238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:30.821339 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:30.821461 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:30.822737 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.823392 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.824879 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.825309 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.825449 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.828902 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:30.828906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:30.828908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:30.828935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:30.830644 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.831272 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.831278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.831283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.831390 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.831680 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.831682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.831683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.831900 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:30.831977 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:30.832500 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 16:20:30.832607 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:20:30.833698 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.819198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.66:53691)
0929 16:20:30.819700 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:30.819760 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:30.820909 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:30.821642 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:30.821650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:30.822001 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:30.822575 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.822592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.823546 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.823549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.824722 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.824732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.825491 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.825496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.825712 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.827047 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:30.827064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:30.828686 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:30.830779 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.830785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.830800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.831136 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.831469 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.831471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.831473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.831601 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.832022 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:30.832028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:30.832260 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 16:20:30.832449 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 16:20:30.832692 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260929 16:20:30.835067 6967 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36361
I20260929 16:20:30.835229 6967 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:30.835772 6967 heartbeater.cc:511] Master 127.4.200.126:36361 requested a full tablet report, sending...
I20260929 16:20:30.836858 6623 ts_manager.cc:194] Registered new tserver with Master: c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607)
I20260929 16:20:30.837533 6623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:53691
I20260929 16:20:30.838011 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6836
I20260929 16:20:30.838120 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:30.838470 4897 external_mini_cluster.cc:1428] Setting key c0252b215d951e84bbd270ba5eb83897
I20260929 16:20:30.845131 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.837132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:38861 (local address 127.4.200.126:36361)
0929 16:20:30.837241 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:30.837245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:30.837998 (+ 753us) server_negotiation.cc:407] Connection header received
0929 16:20:30.838165 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:30.838170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:30.838222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:30.838316 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:30.838843 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.839409 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.840518 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.840725 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.840808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.842026 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:30.842030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:30.842033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:30.842055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:30.843450 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.843907 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.843911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.843913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.843989 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:30.844292 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:30.844294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:30.844296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:30.844507 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:30.844615 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:30.844859 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:20:30.844977 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:20:30.845139 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:30.837089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.66:38861)
0929 16:20:30.837915 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:30.837928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:30.838037 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:30.838346 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:30.838349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:30.838601 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:30.838712 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.838719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.839538 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.839541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.840382 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:30.840396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:30.840838 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:30.840845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:30.840970 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:30.841563 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:30.841577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:30.841895 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:30.843559 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.843564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.843566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.843806 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.844094 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:30.844099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:30.844101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:30.844187 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:30.844603 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:30.844606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:30.844701 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:20:30.844808 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:20:30.844949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:30.866106 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36361
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:30.976673 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:30.976938 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:30.977023 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:30.981089 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:30.981148 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:30.981189 6976 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:20:30.981202 6976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:20:30.981292 6976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:30.986464 6976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:30.987428 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:30.988387 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:30.998548 6976 server_base.cc:1034] running on GCE node
W20260929 16:20:30.998518 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.998576 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:30.998772 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:30.999271 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:30.999974 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:31.001168 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790698831001149 us; error 31 us; skew 500 ppm
Sep 29 16:20:31 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:31.004077 6976 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:31.004959 6976 webserver.cc:460] Webserver started at http://127.4.200.67:35627/ using document root <none> and password file <none>
I20260929 16:20:31.005537 6976 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:31.005627 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:31.005858 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:31.008116 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "a116ca9e667848e7a7dc6c340183208a"
format_stamp: "Formatted at 2026-09-29 16:20:31 on dist-test-slave-s9ct"
server_key: "a9ad4443f182fae25366b42746640d8b"
server_key_iv: "ce60656ca2c69aaefbba99458bdd1466"
server_key_version: "encryptionkey@0"
I20260929 16:20:31.008713 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "a116ca9e667848e7a7dc6c340183208a"
format_stamp: "Formatted at 2026-09-29 16:20:31 on dist-test-slave-s9ct"
server_key: "a9ad4443f182fae25366b42746640d8b"
server_key_iv: "ce60656ca2c69aaefbba99458bdd1466"
server_key_version: "encryptionkey@0"
I20260929 16:20:31.012548 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:20:31.014921 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:31.016124 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:31.016266 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "a116ca9e667848e7a7dc6c340183208a"
format_stamp: "Formatted at 2026-09-29 16:20:31 on dist-test-slave-s9ct"
server_key: "a9ad4443f182fae25366b42746640d8b"
server_key_iv: "ce60656ca2c69aaefbba99458bdd1466"
server_key_version: "encryptionkey@0"
I20260929 16:20:31.016409 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:31.040611 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:31.043896 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:31.044131 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:31.045629 6976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:31.045678 6976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:31.045720 6976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:31.045774 6976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:20:31 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:31.056701 6976 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:43039
I20260929 16:20:31.056728 7106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:43039 every 8 connection(s)
I20260929 16:20:31.057863 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:31.060935 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.046658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:35299 (local address 127.4.200.126:36361)
0929 16:20:31.046819 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.046825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.047818 (+ 993us) server_negotiation.cc:407] Connection header received
0929 16:20:31.049287 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.049293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.049361 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.049484 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:31.051220 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.051848 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.053173 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.053527 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.053639 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.056275 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:31.056279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:31.056282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:31.056319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:31.058338 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.059167 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.059173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.059175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.059276 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.059696 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.059701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.059704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.059953 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:31.060057 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.060612 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.060762 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:20:31.061694 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.047122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.67:35299)
0929 16:20:31.047622 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.047683 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.048967 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.049688 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.049697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.050241 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:31.051039 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.051064 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.051981 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.051985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.053007 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.053038 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.053672 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.053676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.053873 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.054517 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:31.054540 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:31.056091 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:31.058538 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.058545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.058561 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.058938 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.059373 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.059378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.059380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.059555 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.060062 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:31.060067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:31.060340 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.060549 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.060889 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260929 16:20:31.062676 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 6976
I20260929 16:20:31.062791 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:31.062976 7107 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36361
I20260929 16:20:31.063117 7107 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:31.063122 4897 external_mini_cluster.cc:1428] Setting key 83876e69dba8d0c8794c9e0d6c4e27a1
I20260929 16:20:31.063637 7107 heartbeater.cc:511] Master 127.4.200.126:36361 requested a full tablet report, sending...
I20260929 16:20:31.065227 6623 ts_manager.cc:194] Registered new tserver with Master: a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039)
I20260929 16:20:31.065919 6623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:35299
I20260929 16:20:31.073143 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.065522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:48931 (local address 127.4.200.126:36361)
0929 16:20:31.065640 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.065644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.066361 (+ 717us) server_negotiation.cc:407] Connection header received
0929 16:20:31.066526 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.066529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.066579 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.066642 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:31.067154 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.067658 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.068529 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.068736 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.068819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.069883 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:31.069886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:31.069888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:31.069910 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:31.071489 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.072013 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.072018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.072023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.072108 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.072397 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.072399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.072400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.072557 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:31.072629 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.072938 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.073029 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:20:31.073153 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.065458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.4.200.67:48931)
0929 16:20:31.066277 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.066289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.066374 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.066682 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.066685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.066923 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:31.067035 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.067042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.067786 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.067790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.068417 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.068426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.068879 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.068882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.068960 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.069428 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:31.069439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:31.069760 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:31.071585 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.071588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.071589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.071908 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.072211 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.072214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.072215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.072295 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.072644 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:31.072647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:31.072760 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.072887 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.073002 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":97,"threads_started":1}
I20260929 16:20:31.077085 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:20:31 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:20:31 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:31.103775 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.094855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.4.200.126:36361)
0929 16:20:31.094989 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.094993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.095005 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:31.095045 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.095047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.095085 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.095149 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:31.095926 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.096473 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.097463 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.097699 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.097822 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.100615 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:31.100618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:31.100620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:31.100650 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:31.101954 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.102439 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.102445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.102451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.102536 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.102824 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.102826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.102828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.103005 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:31.103105 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.103454 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.103553 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:20:31.106896 6623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:31.110198 6623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:31.124894 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.119193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.4.200.66:35607)
0929 16:20:31.119826 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.119842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.119869 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:20:31.119947 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.119952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.120174 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.120358 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:31.120867 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.121934 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.123417 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.124181 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.124333 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.124519 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.124639 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.124728 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":112,"threads_started":1}
I20260929 16:20:31.124878 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.119022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35607 (local address 127.0.0.1:38382)
0929 16:20:31.119417 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.119472 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.119598 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.120554 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.120560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.120584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:31.120734 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.120741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.122058 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.122062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.123277 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.123288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.124312 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.124315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.124448 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.124489 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.124609 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.124729 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
I20260929 16:20:31.125770 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.119274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:37991 (local address 127.0.0.1:52564)
0929 16:20:31.120238 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.120255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.120378 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.121128 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.121131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.121151 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:31.121244 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.121250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.122615 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.122619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.124407 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.124414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.125405 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.125408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.125498 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.125521 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.125584 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.125646 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260929 16:20:31.125857 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.119895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.4.200.65:37991)
0929 16:20:31.120632 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.120639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.120662 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:20:31.120742 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.120747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.120943 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.121150 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:31.121369 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.122478 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.124524 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.125259 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.125419 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.125572 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.125622 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.125711 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":87,"threads_started":1}
I20260929 16:20:31.125895 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.120093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.0.0.1:36828)
0929 16:20:31.120334 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.120346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.120419 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.121814 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.121818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.121836 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:31.121963 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.121974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.123283 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.123286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.124231 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.124240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.125480 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.125484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.125583 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.125611 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.125658 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.125723 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":31,"thread_start_us":105,"threads_started":1}
I20260929 16:20:31.126549 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.120613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.4.200.67:43039)
0929 16:20:31.121355 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.121367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.121388 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:20:31.121458 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.121463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.121655 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.121801 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:31.122101 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.123158 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.124358 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.125344 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.126117 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.126261 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.126304 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.126418 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":97,"threads_started":1}
I20260929 16:20:31.128130 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 66f941aebd70445abccbf67657e1c143 (DEFAULT_TABLE table=test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.128530 6761 tablet_service.cc:1467] Processing CreateTablet for tablet 66f941aebd70445abccbf67657e1c143 (DEFAULT_TABLE table=test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.128763 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 66f941aebd70445abccbf67657e1c143 (DEFAULT_TABLE table=test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.129135 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f941aebd70445abccbf67657e1c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.129541 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f941aebd70445abccbf67657e1c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.129536 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f941aebd70445abccbf67657e1c143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.136451 7130 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Bootstrap starting.
I20260929 16:20:31.137215 7131 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Bootstrap starting.
I20260929 16:20:31.137389 7132 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Bootstrap starting.
I20260929 16:20:31.139207 7132 tablet_bootstrap.cc:654] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.139292 7130 tablet_bootstrap.cc:654] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.139299 7131 tablet_bootstrap.cc:654] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.139954 7132 log.cc:824] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:31.140098 7131 log.cc:824] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:31.140098 7130 log.cc:824] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:31.141957 7130 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: No bootstrap required, opened a new log
I20260929 16:20:31.142015 7132 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: No bootstrap required, opened a new log
I20260929 16:20:31.142015 7131 tablet_bootstrap.cc:492] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: No bootstrap required, opened a new log
I20260929 16:20:31.142174 7130 ts_tablet_manager.cc:1397] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:20:31.142177 7132 ts_tablet_manager.cc:1397] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:20:31.142177 7131 ts_tablet_manager.cc:1397] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:20:31.145491 7130 raft_consensus.cc:348] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.145735 7130 raft_consensus.cc:374] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.145785 7130 raft_consensus.cc:729] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.146090 7131 raft_consensus.cc:348] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.146090 7132 raft_consensus.cc:348] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.146229 7130 consensus_queue.cc:260] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.146340 7131 raft_consensus.cc:374] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.146340 7132 raft_consensus.cc:374] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.146420 7132 raft_consensus.cc:729] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.146428 7131 raft_consensus.cc:729] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.146874 6827 heartbeater.cc:503] Master 127.4.200.126:36361 was elected leader, sending a full tablet report...
I20260929 16:20:31.146826 7131 consensus_queue.cc:260] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.147298 7130 ts_tablet_manager.cc:1428] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:20:31.147555 6967 heartbeater.cc:503] Master 127.4.200.126:36361 was elected leader, sending a full tablet report...
I20260929 16:20:31.148056 7131 ts_tablet_manager.cc:1428] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:20:31.148097 7132 consensus_queue.cc:260] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.148813 7107 heartbeater.cc:503] Master 127.4.200.126:36361 was elected leader, sending a full tablet report...
I20260929 16:20:31.149056 7132 ts_tablet_manager.cc:1428] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260929 16:20:31.152809 7138 raft_consensus.cc:482] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:31.152998 7138 raft_consensus.cc:504] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.153975 7138 leader_election.cc:288] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607), f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991)
I20260929 16:20:31.158200 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.154242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:45255 (local address 127.4.200.66:35607)
0929 16:20:31.154526 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.154530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.154545 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:20:31.154593 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.154597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.154641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.154730 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:31.155101 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.155707 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.157055 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.157668 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.157796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.157852 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.158050 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.158105 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260929 16:20:31.158231 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.154197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35607 (local address 127.4.200.67:45255)
0929 16:20:31.154324 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.154340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.154459 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.154813 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.154816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.154841 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:31.154981 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.154992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.155836 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.155840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.156920 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.156935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.157805 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.157816 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.157925 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.157951 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.158037 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.158105 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260929 16:20:31.159014 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.154206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:37991 (local address 127.4.200.67:42261)
0929 16:20:31.154709 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.154724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.154812 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.155100 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.155102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.155118 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:31.155232 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.155238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.156037 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.156040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.157672 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.157680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.158619 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.158622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.158690 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.158708 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.158810 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.158874 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":142,"thread_start_us":165,"threads_started":1}
I20260929 16:20:31.159020 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.154365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:42261 (local address 127.4.200.65:37991)
0929 16:20:31.154503 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.154508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.154818 (+ 310us) server_negotiation.cc:407] Connection header received
0929 16:20:31.154908 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.154911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.154963 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.155056 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:31.155342 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.155902 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.157792 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.158491 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.158601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.158635 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.158825 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.158882 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:20:31.159173 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f941aebd70445abccbf67657e1c143" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" is_pre_election: true
I20260929 16:20:31.159475 6921 raft_consensus.cc:2454] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.159577 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f941aebd70445abccbf67657e1c143" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1" is_pre_election: true
I20260929 16:20:31.159825 6781 raft_consensus.cc:2454] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.159956 6994 leader_election.cc:302] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, c8dcba06e9d144b98a36b1c1cf422ee5; no voters:
I20260929 16:20:31.160220 7138 raft_consensus.cc:2781] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:31.160301 7138 raft_consensus.cc:482] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:31.160351 7138 raft_consensus.cc:3037] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.161362 7138 raft_consensus.cc:504] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.161695 7138 leader_election.cc:288] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Requested vote from peers c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607), f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991)
I20260929 16:20:31.161954 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f941aebd70445abccbf67657e1c143" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
I20260929 16:20:31.162055 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f941aebd70445abccbf67657e1c143" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1"
I20260929 16:20:31.162079 6921 raft_consensus.cc:3037] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.162196 6781 raft_consensus.cc:3037] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.163316 6781 raft_consensus.cc:2454] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.163300 6921 raft_consensus.cc:2454] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.163602 6992 leader_election.cc:302] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:31.163793 7138 raft_consensus.cc:2781] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:31.164088 7138 raft_consensus.cc:686] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 1 LEADER]: Becoming Leader. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Running, Role: LEADER
I20260929 16:20:31.164427 7138 consensus_queue.cc:237] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.167698 6624 catalog_manager.cc:5495] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a reported cstate change: term changed from 0 to 1, leader changed from <none> to a116ca9e667848e7a7dc6c340183208a (127.4.200.67). New cstate: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } health_report { overall_health: HEALTHY } } }
Sep 29 16:20:31 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:31.220685 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.212345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.4.200.126:36361)
0929 16:20:31.212530 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.212534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.212637 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:20:31.212789 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.212792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.212835 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.212922 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:31.213596 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.214133 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.214919 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.215118 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.215194 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.217397 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:31.217400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:31.217402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:31.217420 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:31.218874 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.219516 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.219518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.219519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.219590 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:31.219826 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:31.219828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:31.219829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:31.220019 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:31.220115 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.220465 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.220555 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 16:20:31.220688 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.212230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36361 (local address 127.0.0.1:56176)
0929 16:20:31.212547 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.212561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.212673 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.212967 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.212969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.213372 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:31.213474 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.213480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.214227 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.214229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.214817 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.214825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.215217 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.215219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.215297 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.215854 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:31.215866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:31.217269 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:31.219106 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.219114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.219119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.219430 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.219670 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:31.219673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:31.219674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:31.219753 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:31.220129 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:31.220134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:31.220232 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.220386 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.220503 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260929 16:20:31.224534 6624 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:56176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:20:31.225286 6624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:31.230067 6761 tablet_service.cc:1467] Processing CreateTablet for tablet f60733e133f84e48a7745bf9eb18f6e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cba4e050a1b4dffa84fca5a2276a9c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:31.230358 6901 tablet_service.cc:1467] Processing CreateTablet for tablet f60733e133f84e48a7745bf9eb18f6e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cba4e050a1b4dffa84fca5a2276a9c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:31.230268 7041 tablet_service.cc:1467] Processing CreateTablet for tablet f60733e133f84e48a7745bf9eb18f6e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cba4e050a1b4dffa84fca5a2276a9c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:20:31.230888 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60733e133f84e48a7745bf9eb18f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.230933 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60733e133f84e48a7745bf9eb18f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.230974 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60733e133f84e48a7745bf9eb18f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.233613 7132 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: Bootstrap starting.
I20260929 16:20:31.233716 7131 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: Bootstrap starting.
I20260929 16:20:31.233865 7130 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: Bootstrap starting.
I20260929 16:20:31.234699 7132 tablet_bootstrap.cc:654] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.234701 7131 tablet_bootstrap.cc:654] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.234833 7130 tablet_bootstrap.cc:654] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.236032 7132 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: No bootstrap required, opened a new log
I20260929 16:20:31.236032 7131 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: No bootstrap required, opened a new log
I20260929 16:20:31.236111 7132 ts_tablet_manager.cc:1397] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:31.236111 7131 ts_tablet_manager.cc:1397] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:31.236398 7130 tablet_bootstrap.cc:492] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: No bootstrap required, opened a new log
I20260929 16:20:31.236455 7130 ts_tablet_manager.cc:1397] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:31.236537 7132 raft_consensus.cc:348] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.236537 7131 raft_consensus.cc:348] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.236645 7131 raft_consensus.cc:374] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.236645 7132 raft_consensus.cc:374] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.236680 7132 raft_consensus.cc:729] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.236680 7131 raft_consensus.cc:729] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.236804 7131 consensus_queue.cc:260] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.236804 7132 consensus_queue.cc:260] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.236817 7130 raft_consensus.cc:348] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.236961 7130 raft_consensus.cc:374] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.236999 7130 raft_consensus.cc:729] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.237061 7132 ts_tablet_manager.cc:1428] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:31.237061 7131 ts_tablet_manager.cc:1428] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:20:31.237082 7137 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.237118 7130 consensus_queue.cc:260] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.237212 7137 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.237241 7138 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.237363 7138 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.237387 7130 ts_tablet_manager.cc:1428] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:31.237401 7136 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.238116 7136 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 16:20:31.309201 7108 tablet.cc:2369] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:20:31.331189 6968 tablet.cc:2369] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:20:31.355202 6828 tablet.cc:2369] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:31.473255 7138 raft_consensus.cc:482] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:31.473415 7138 raft_consensus.cc:504] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.473973 7138 leader_election.cc:288] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607)
I20260929 16:20:31.474547 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60733e133f84e48a7745bf9eb18f6e6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" is_pre_election: true
I20260929 16:20:31.474548 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60733e133f84e48a7745bf9eb18f6e6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1" is_pre_election: true
I20260929 16:20:31.474711 6781 raft_consensus.cc:2454] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.474711 6921 raft_consensus.cc:2454] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.475046 6994 leader_election.cc:302] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, c8dcba06e9d144b98a36b1c1cf422ee5, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:31.475204 7138 raft_consensus.cc:2781] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:31.475266 7138 raft_consensus.cc:482] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:31.475309 7138 raft_consensus.cc:3037] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.476308 7138 raft_consensus.cc:504] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.476647 7138 leader_election.cc:288] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Requested vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607)
I20260929 16:20:31.477002 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60733e133f84e48a7745bf9eb18f6e6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1"
I20260929 16:20:31.477002 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60733e133f84e48a7745bf9eb18f6e6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
I20260929 16:20:31.477126 6781 raft_consensus.cc:3037] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.477126 6921 raft_consensus.cc:3037] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.478226 6781 raft_consensus.cc:2454] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.478226 6921 raft_consensus.cc:2454] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.478567 6992 leader_election.cc:302] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, c8dcba06e9d144b98a36b1c1cf422ee5, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:31.478783 7138 raft_consensus.cc:2781] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:31.478912 7138 raft_consensus.cc:686] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [term 1 LEADER]: Becoming Leader. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Running, Role: LEADER
I20260929 16:20:31.479039 7138 consensus_queue.cc:237] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.479734 7140 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: TxnStatusTablet state changed. Reason: New leader a116ca9e667848e7a7dc6c340183208a. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.479843 7140 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:31.480161 7152 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:20:31.480410 7152 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:20:31.480865 6623 catalog_manager.cc:5495] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a reported cstate change: term changed from 0 to 1, leader changed from <none> to a116ca9e667848e7a7dc6c340183208a (127.4.200.67). New cstate: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } health_report { overall_health: HEALTHY } } }
I20260929 16:20:31.533005 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.528626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.0.0.1:36834)
0929 16:20:31.528785 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.528808 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.528896 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.529270 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.529273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.529280 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:31.529379 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.529384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.530474 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.530477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.531084 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.531093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.531677 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.531681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.531765 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.531812 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:31.532609 (+ 797us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:31.532626 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.532762 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.532878 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:20:31.533011 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.528757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.4.200.67:43039)
0929 16:20:31.528872 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.528877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.528892 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:20:31.529036 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.529039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.529104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.529214 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:31.529496 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.530292 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.531197 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.531544 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.531647 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.531946 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:31.532495 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:31.532612 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.532761 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.532826 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:20:31.539029 6781 raft_consensus.cc:1261] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.539057 6921 raft_consensus.cc:1261] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.539706 7140 consensus_queue.cc:1035] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:31.539916 7138 consensus_queue.cc:1035] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:31.543429 7136 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: TxnStatusTablet state changed. Reason: New leader a116ca9e667848e7a7dc6c340183208a. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.543579 7136 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.543782 7137 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: TxnStatusTablet state changed. Reason: New leader a116ca9e667848e7a7dc6c340183208a. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.543895 7137 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.546566 7140 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.546684 7140 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:31.548645 7136 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.548774 7136 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P f808264d9be546ae8ce4e9f5526d80c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.549479 7138 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.549593 7138 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P a116ca9e667848e7a7dc6c340183208a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:20:31.550642 7137 tablet_replica.cc:439] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } } }
I20260929 16:20:31.550750 7137 tablet_replica.cc:442] T f60733e133f84e48a7745bf9eb18f6e6 P c8dcba06e9d144b98a36b1c1cf422ee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:20:31.551913 7157 mvcc.cc:205] Tried to move back new op lower bound from 7334702413976989696 to 7334702413739327488. Current Snapshot: MvccSnapshot[applied={T|T < 7334702413976989696}]
I20260929 16:20:31.552930 7156 mvcc.cc:205] Tried to move back new op lower bound from 7334702413976989696 to 7334702413739327488. Current Snapshot: MvccSnapshot[applied={T|T < 7334702413976989696}]
I20260929 16:20:31.558724 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.555115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.4.200.67:43039)
0929 16:20:31.555219 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.555224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.555237 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:20:31.555284 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.555286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.555345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.555431 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:31.555829 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.556377 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.557622 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.557834 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.557930 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.558210 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:31.558294 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:31.558365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.558519 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.558567 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:20:31.570654 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.567239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:46371 (local address 127.4.200.67:43039)
0929 16:20:31.567355 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:31.567358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:31.567449 (+ 91us) server_negotiation.cc:407] Connection header received
0929 16:20:31.567582 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:31.567584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:31.567622 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:31.567684 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:31.567943 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.568422 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.569611 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.569816 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.569916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.570204 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:31.570269 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:31.570322 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:31.570493 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 16:20:31.570535 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260929 16:20:31.570689 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:31.567174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.4.200.67:46371)
0929 16:20:31.567393 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:31.567407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:31.567501 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:31.567714 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:31.567717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:31.567724 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:31.567846 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.567853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.568520 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.568523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.569462 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:31.569472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:31.569927 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:31.569931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:31.570005 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:31.570053 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:31.570346 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:31.570352 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:20:31.570472 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:20:31.570548 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":18}
I20260929 16:20:31.577672 6921 raft_consensus.cc:1261] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.577788 6781 raft_consensus.cc:1261] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.578166 7138 consensus_queue.cc:1035] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:31.578343 7140 consensus_queue.cc:1035] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:31.584033 7155 mvcc.cc:205] Tried to move back new op lower bound from 7334702414138925056 to 7334702412452204544. Current Snapshot: MvccSnapshot[applied={T|T < 7334702414138925056}]
I20260929 16:20:31.626225 6622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:31.626446 6622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:31.628780 6622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Sending DeleteTablet for 3 replicas of tablet 66f941aebd70445abccbf67657e1c143
I20260929 16:20:31.629524 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f941aebd70445abccbf67657e1c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:38382
I20260929 16:20:31.629524 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f941aebd70445abccbf67657e1c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260929 16:20:31.630033 7170 tablet_replica.cc:330] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: stopping tablet replica
I20260929 16:20:31.630102 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f941aebd70445abccbf67657e1c143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:36828
I20260929 16:20:31.630484 7170 raft_consensus.cc:2229] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:31.630506 7171 tablet_replica.cc:330] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: stopping tablet replica
I20260929 16:20:31.630738 7170 raft_consensus.cc:2258] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:31.630698 6622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:20:31.630792 7171 raft_consensus.cc:2229] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:31.630980 7171 raft_consensus.cc:2258] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:20:31.631096 6622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:31.632356 7171 ts_tablet_manager.cc:1905] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:31.633394 7170 ts_tablet_manager.cc:1905] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:31.635987 6761 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 (DEFAULT_TABLE table=test-table [id=95508f0219e04bffa9eabe59e8b65ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.636245 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8731c6f1be4dbc8f35875e552656d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.636350 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 (DEFAULT_TABLE table=test-table [id=95508f0219e04bffa9eabe59e8b65ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.636330 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 (DEFAULT_TABLE table=test-table [id=95508f0219e04bffa9eabe59e8b65ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:31.636596 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8731c6f1be4dbc8f35875e552656d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.636633 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8731c6f1be4dbc8f35875e552656d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:31.638497 7130 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Bootstrap starting.
I20260929 16:20:31.639017 7131 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Bootstrap starting.
I20260929 16:20:31.639422 7130 tablet_bootstrap.cc:654] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.639918 7131 tablet_bootstrap.cc:654] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.640525 7132 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Bootstrap starting.
I20260929 16:20:31.640692 7130 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: No bootstrap required, opened a new log
I20260929 16:20:31.640758 7130 ts_tablet_manager.cc:1397] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:31.641005 7131 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: No bootstrap required, opened a new log
I20260929 16:20:31.641070 7131 ts_tablet_manager.cc:1397] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:31.641188 7130 raft_consensus.cc:348] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.641291 7130 raft_consensus.cc:374] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.641353 7130 raft_consensus.cc:729] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.641403 7132 tablet_bootstrap.cc:654] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:31.641472 7131 raft_consensus.cc:348] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.641484 7130 consensus_queue.cc:260] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.641569 7131 raft_consensus.cc:374] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.641597 7131 raft_consensus.cc:729] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.641709 7131 consensus_queue.cc:260] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.641795 7130 ts_tablet_manager.cc:1428] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:31.641937 7131 ts_tablet_manager.cc:1428] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:20:31.642534 7172 tablet_replica.cc:330] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: stopping tablet replica
I20260929 16:20:31.642880 7172 raft_consensus.cc:2229] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:31.643203 7172 raft_consensus.cc:2258] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:31.644263 7172 ts_tablet_manager.cc:1905] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:31.645565 7170 ts_tablet_manager.cc:1918] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:31.645668 7170 log.cc:1196] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/66f941aebd70445abccbf67657e1c143
I20260929 16:20:31.646006 7170 ts_tablet_manager.cc:1939] T 66f941aebd70445abccbf67657e1c143 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting consensus metadata
I20260929 16:20:31.646337 7171 ts_tablet_manager.cc:1918] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:31.646457 7171 log.cc:1196] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/66f941aebd70445abccbf67657e1c143
I20260929 16:20:31.646775 7171 ts_tablet_manager.cc:1939] T 66f941aebd70445abccbf67657e1c143 P f808264d9be546ae8ce4e9f5526d80c1: Deleting consensus metadata
I20260929 16:20:31.646891 7172 ts_tablet_manager.cc:1918] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:31.647003 7172 log.cc:1196] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/66f941aebd70445abccbf67657e1c143
I20260929 16:20:31.647310 7172 ts_tablet_manager.cc:1939] T 66f941aebd70445abccbf67657e1c143 P a116ca9e667848e7a7dc6c340183208a: Deleting consensus metadata
I20260929 16:20:31.647343 6610 catalog_manager.cc:4841] TS c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607): tablet 66f941aebd70445abccbf67657e1c143 (table test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]) successfully deleted
I20260929 16:20:31.647863 7132 tablet_bootstrap.cc:492] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: No bootstrap required, opened a new log
I20260929 16:20:31.647964 7132 ts_tablet_manager.cc:1397] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 16:20:31.647929 6608 catalog_manager.cc:4841] TS f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991): tablet 66f941aebd70445abccbf67657e1c143 (table test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]) successfully deleted
I20260929 16:20:31.648201 6611 catalog_manager.cc:4841] TS a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039): tablet 66f941aebd70445abccbf67657e1c143 (table test-table [id=67e24f0e992d4e218b93bc9d0ef0e7b5]) successfully deleted
I20260929 16:20:31.648397 7132 raft_consensus.cc:348] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.648545 7132 raft_consensus.cc:374] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:31.648590 7132 raft_consensus.cc:729] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Initialized, Role: FOLLOWER
I20260929 16:20:31.648715 7132 consensus_queue.cc:260] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.649009 7132 ts_tablet_manager.cc:1428] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:20:31.675017 7140 raft_consensus.cc:482] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:31.675150 7140 raft_consensus.cc:504] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.675639 7140 leader_election.cc:288] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607)
I20260929 16:20:31.676023 6780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8731c6f1be4dbc8f35875e552656d6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1" is_pre_election: true
I20260929 16:20:31.676023 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8731c6f1be4dbc8f35875e552656d6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" is_pre_election: true
I20260929 16:20:31.676180 6920 raft_consensus.cc:2454] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.676180 6780 raft_consensus.cc:2454] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 0.
I20260929 16:20:31.676471 6992 leader_election.cc:302] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:31.676635 7140 raft_consensus.cc:2781] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:31.676700 7140 raft_consensus.cc:482] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:31.676764 7140 raft_consensus.cc:3037] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.677747 7140 raft_consensus.cc:504] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.678125 7140 leader_election.cc:288] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Requested vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607)
I20260929 16:20:31.678439 6780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8731c6f1be4dbc8f35875e552656d6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1"
I20260929 16:20:31.678508 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8731c6f1be4dbc8f35875e552656d6" candidate_uuid: "a116ca9e667848e7a7dc6c340183208a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
I20260929 16:20:31.678583 6780 raft_consensus.cc:3037] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.678648 6920 raft_consensus.cc:3037] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:31.679494 6780 raft_consensus.cc:2454] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.679774 6920 raft_consensus.cc:2454] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a116ca9e667848e7a7dc6c340183208a in term 1.
I20260929 16:20:31.679780 6992 leader_election.cc:302] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:31.679968 7140 raft_consensus.cc:2781] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:31.680058 7140 raft_consensus.cc:686] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 1 LEADER]: Becoming Leader. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Running, Role: LEADER
I20260929 16:20:31.680209 7140 consensus_queue.cc:237] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:31.681766 6623 catalog_manager.cc:5495] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a reported cstate change: term changed from 0 to 1, leader changed from <none> to a116ca9e667848e7a7dc6c340183208a (127.4.200.67). New cstate: current_term: 1 leader_uuid: "a116ca9e667848e7a7dc6c340183208a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } health_report { overall_health: HEALTHY } } }
I20260929 16:20:31.706205 6920 raft_consensus.cc:1261] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.706451 6780 raft_consensus.cc:1261] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Refusing update from remote peer a116ca9e667848e7a7dc6c340183208a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:31.706652 7140 consensus_queue.cc:1035] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:31.706876 7138 consensus_queue.cc:1035] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:20:31.722354 7155 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:20:33.274525 6624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:33.274714 6624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:33.277173 6624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Sending DeleteTablet for 3 replicas of tablet 7e8731c6f1be4dbc8f35875e552656d6
I20260929 16:20:33.277899 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:38382
I20260929 16:20:33.277922 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260929 16:20:33.278010 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8731c6f1be4dbc8f35875e552656d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:36828
I20260929 16:20:33.278255 7199 tablet_replica.cc:330] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: stopping tablet replica
I20260929 16:20:33.278616 7199 raft_consensus.cc:2229] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:33.278757 7199 raft_consensus.cc:2258] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.279465 7199 ts_tablet_manager.cc:1905] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.281666 7198 tablet_replica.cc:330] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: stopping tablet replica
I20260929 16:20:33.281832 7198 raft_consensus.cc:2229] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:33.281865 7199 ts_tablet_manager.cc:1918] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:20:33.281935 7199 log.cc:1196] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/7e8731c6f1be4dbc8f35875e552656d6
I20260929 16:20:33.281977 7200 tablet_replica.cc:330] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: stopping tablet replica
I20260929 16:20:33.282064 7198 raft_consensus.cc:2258] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.282153 7200 raft_consensus.cc:2229] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:33.282371 7199 ts_tablet_manager.cc:1939] T 7e8731c6f1be4dbc8f35875e552656d6 P f808264d9be546ae8ce4e9f5526d80c1: Deleting consensus metadata
I20260929 16:20:33.282442 7200 raft_consensus.cc:2258] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.282853 7198 ts_tablet_manager.cc:1905] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.283078 7200 ts_tablet_manager.cc:1905] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.283223 6608 catalog_manager.cc:4841] TS f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991): tablet 7e8731c6f1be4dbc8f35875e552656d6 (table test-table [id=95508f0219e04bffa9eabe59e8b65ada]) successfully deleted
I20260929 16:20:33.285245 7200 ts_tablet_manager.cc:1918] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:20:33.285341 7200 log.cc:1196] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/7e8731c6f1be4dbc8f35875e552656d6
I20260929 16:20:33.285620 7200 ts_tablet_manager.cc:1939] T 7e8731c6f1be4dbc8f35875e552656d6 P a116ca9e667848e7a7dc6c340183208a: Deleting consensus metadata
I20260929 16:20:33.286024 7198 ts_tablet_manager.cc:1918] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:20:33.286134 7198 log.cc:1196] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/7e8731c6f1be4dbc8f35875e552656d6
I20260929 16:20:33.286221 6611 catalog_manager.cc:4841] TS a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039): tablet 7e8731c6f1be4dbc8f35875e552656d6 (table test-table [id=95508f0219e04bffa9eabe59e8b65ada]) successfully deleted
I20260929 16:20:33.286463 7198 ts_tablet_manager.cc:1939] T 7e8731c6f1be4dbc8f35875e552656d6 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting consensus metadata
I20260929 16:20:33.287217 6610 catalog_manager.cc:4841] TS c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607): tablet 7e8731c6f1be4dbc8f35875e552656d6 (table test-table [id=95508f0219e04bffa9eabe59e8b65ada]) successfully deleted
Sep 29 16:20:33 dist-test-slave-s9ct krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:20:33 dist-test-slave-s9ct krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:33.306308 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.297828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.4.200.126:36361)
0929 16:20:33.298115 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.298126 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.298218 (+ 92us) server_negotiation.cc:407] Connection header received
0929 16:20:33.298439 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.298442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.298523 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.298618 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:33.299309 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.299865 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.300703 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.300949 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.301060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.303420 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:33.303423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:33.303426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:33.303450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:33.304838 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:33.305219 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:33.305222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:33.305223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:33.305294 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:33.305540 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:33.305542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:33.305543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:33.305703 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:33.305801 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.306057 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.306150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260929 16:20:33.309026 6624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:33.309489 6624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:33.314121 6761 tablet_service.cc:1467] Processing CreateTablet for tablet e0ec591f6a444b79a987f6164e4acf27 (DEFAULT_TABLE table=test-table [id=311a76130c0f415fab2663c74d67497f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.314316 7041 tablet_service.cc:1467] Processing CreateTablet for tablet e0ec591f6a444b79a987f6164e4acf27 (DEFAULT_TABLE table=test-table [id=311a76130c0f415fab2663c74d67497f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.314316 6901 tablet_service.cc:1467] Processing CreateTablet for tablet e0ec591f6a444b79a987f6164e4acf27 (DEFAULT_TABLE table=test-table [id=311a76130c0f415fab2663c74d67497f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.314431 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ec591f6a444b79a987f6164e4acf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.314627 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ec591f6a444b79a987f6164e4acf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.314627 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ec591f6a444b79a987f6164e4acf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.317376 7211 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Bootstrap starting.
I20260929 16:20:33.317792 7210 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Bootstrap starting.
I20260929 16:20:33.318333 7211 tablet_bootstrap.cc:654] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.318753 7210 tablet_bootstrap.cc:654] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.319599 7211 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: No bootstrap required, opened a new log
I20260929 16:20:33.319729 7211 ts_tablet_manager.cc:1397] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:33.319878 7210 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: No bootstrap required, opened a new log
I20260929 16:20:33.319984 7210 ts_tablet_manager.cc:1397] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:33.320281 7211 raft_consensus.cc:348] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.320396 7211 raft_consensus.cc:374] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.320456 7211 raft_consensus.cc:729] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.320405 7210 raft_consensus.cc:348] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.320511 7210 raft_consensus.cc:374] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.320550 7210 raft_consensus.cc:729] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.320602 7211 consensus_queue.cc:260] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.320673 7210 consensus_queue.cc:260] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.320963 7210 ts_tablet_manager.cc:1428] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:20:33.321075 7211 ts_tablet_manager.cc:1428] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:33.323344 7209 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Bootstrap starting.
I20260929 16:20:33.324651 7209 tablet_bootstrap.cc:654] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.325699 7209 tablet_bootstrap.cc:492] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: No bootstrap required, opened a new log
I20260929 16:20:33.325769 7209 ts_tablet_manager.cc:1397] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:33.326182 7209 raft_consensus.cc:348] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.326301 7209 raft_consensus.cc:374] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.326354 7209 raft_consensus.cc:729] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.326512 7209 consensus_queue.cc:260] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.326891 7209 ts_tablet_manager.cc:1428] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:20:33.462568 7214 raft_consensus.cc:482] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:33.462764 7214 raft_consensus.cc:504] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.464125 7214 leader_election.cc:288] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039)
I20260929 16:20:33.468331 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.464351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:37991 (local address 127.4.200.66:38163)
0929 16:20:33.464745 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.464760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.464889 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.465190 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.465192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.465208 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:33.465414 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.465420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.466261 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.466263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.467150 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.467158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.467869 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.467975 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.468020 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.468099 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.468167 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":29,"spinlock_wait_cycles":54016,"thread_start_us":137,"threads_started":1}
I20260929 16:20:33.468384 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.464430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:38163 (local address 127.4.200.65:37991)
0929 16:20:33.464769 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.464778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.464826 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:20:33.465025 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.465028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.465081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.465148 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:33.465531 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.466100 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467276 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.467723 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467880 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.467942 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.468144 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.468232 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260929 16:20:33.468608 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.464463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:45267 (local address 127.4.200.67:43039)
0929 16:20:33.464747 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.464760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.464884 (+ 124us) server_negotiation.cc:407] Connection header received
0929 16:20:33.465048 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.465050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.465089 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.465187 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:33.465568 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.466079 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467283 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.467729 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467855 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.467921 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.468385 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.468470 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260929 16:20:33.468854 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.464358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.4.200.66:45267)
0929 16:20:33.464783 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.464798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.464917 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.465211 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.465214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.465242 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:33.465431 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.465437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.466190 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.466196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.467148 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.467157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.467982 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.467989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.468141 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.468165 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.468233 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.468298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":125,"thread_start_us":108,"threads_started":1}
I20260929 16:20:33.468937 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ec591f6a444b79a987f6164e4acf27" candidate_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1" is_pre_election: true
I20260929 16:20:33.469085 6781 raft_consensus.cc:2454] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8dcba06e9d144b98a36b1c1cf422ee5 in term 0.
I20260929 16:20:33.469615 6852 leader_election.cc:302] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8dcba06e9d144b98a36b1c1cf422ee5, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:33.469650 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ec591f6a444b79a987f6164e4acf27" candidate_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a116ca9e667848e7a7dc6c340183208a" is_pre_election: true
I20260929 16:20:33.469858 7061 raft_consensus.cc:2454] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8dcba06e9d144b98a36b1c1cf422ee5 in term 0.
I20260929 16:20:33.469907 7214 raft_consensus.cc:2781] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:33.469988 7214 raft_consensus.cc:482] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:33.470059 7214 raft_consensus.cc:3037] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.471153 7214 raft_consensus.cc:504] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.471563 7214 leader_election.cc:288] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [CANDIDATE]: Term 1 election: Requested vote from peers f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991), a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039)
I20260929 16:20:33.471864 6781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ec591f6a444b79a987f6164e4acf27" candidate_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f808264d9be546ae8ce4e9f5526d80c1"
I20260929 16:20:33.471977 6781 raft_consensus.cc:3037] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.471936 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ec591f6a444b79a987f6164e4acf27" candidate_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a116ca9e667848e7a7dc6c340183208a"
I20260929 16:20:33.472070 7061 raft_consensus.cc:3037] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.472995 7061 raft_consensus.cc:2454] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8dcba06e9d144b98a36b1c1cf422ee5 in term 1.
I20260929 16:20:33.473097 6781 raft_consensus.cc:2454] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8dcba06e9d144b98a36b1c1cf422ee5 in term 1.
I20260929 16:20:33.473330 6855 leader_election.cc:302] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a116ca9e667848e7a7dc6c340183208a, c8dcba06e9d144b98a36b1c1cf422ee5; no voters:
I20260929 16:20:33.473502 7214 raft_consensus.cc:2781] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:33.473721 7214 raft_consensus.cc:686] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 LEADER]: Becoming Leader. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Running, Role: LEADER
I20260929 16:20:33.473970 7214 consensus_queue.cc:237] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.476485 6622 catalog_manager.cc:5495] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66). New cstate: current_term: 1 leader_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } health_report { overall_health: UNKNOWN } } }
I20260929 16:20:33.520177 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.516287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.4.200.66:35607)
0929 16:20:33.516589 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.516595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.516607 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:33.516675 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.516677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.516727 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.516830 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.517088 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.517623 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.518579 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.518819 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.518897 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.519185 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.519743 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.519830 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.519933 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.519979 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":167,"threads_started":1}
I20260929 16:20:33.530334 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.526607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.4.200.66:32843)
0929 16:20:33.527363 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.527376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.527491 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.527749 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.527752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.527760 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:33.527885 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.527891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.528579 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.528586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.529200 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.529208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.529641 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.529644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.529745 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.529782 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.530065 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.530074 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.530133 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.530206 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":565,"threads_started":1}
I20260929 16:20:33.530320 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.527134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:32843 (local address 127.4.200.67:43039)
0929 16:20:33.527289 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.527293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.527449 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:20:33.527600 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.527608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.527651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.527726 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.527988 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.528475 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.529318 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.529535 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.529644 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.529915 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.529976 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.530051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.530142 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.530187 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:20:33.536158 6781 raft_consensus.cc:1261] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Refusing update from remote peer c8dcba06e9d144b98a36b1c1cf422ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:33.536211 7061 raft_consensus.cc:1261] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Refusing update from remote peer c8dcba06e9d144b98a36b1c1cf422ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:33.536665 7214 consensus_queue.cc:1035] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:33.536870 7221 consensus_queue.cc:1035] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:33.554679 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.551004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:40403 (local address 127.4.200.66:35607)
0929 16:20:33.551132 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.551137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.551430 (+ 293us) server_negotiation.cc:407] Connection header received
0929 16:20:33.551556 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.551559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.551611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.551698 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.552054 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.552561 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.553446 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.553792 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.553935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.554191 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.554272 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.554352 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.554475 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.554533 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:20:33.554685 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.550922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35607 (local address 127.4.200.67:40403)
0929 16:20:33.551324 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.551339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.551446 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.551825 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.551828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.551836 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:33.551923 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.551927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.552678 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.552681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.553285 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.553293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.553931 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.553935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.554041 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.554067 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.554365 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.554374 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.554465 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.554555 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":122,"threads_started":1}
I20260929 16:20:33.580056 6624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:33.580240 6624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:33.582571 6624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Sending DeleteTablet for 3 replicas of tablet e0ec591f6a444b79a987f6164e4acf27
I20260929 16:20:33.583269 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ec591f6a444b79a987f6164e4acf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:38382
I20260929 16:20:33.583288 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ec591f6a444b79a987f6164e4acf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260929 16:20:33.583487 7199 tablet_replica.cc:330] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: stopping tablet replica
I20260929 16:20:33.583487 7198 tablet_replica.cc:330] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: stopping tablet replica
I20260929 16:20:33.583611 7199 raft_consensus.cc:2229] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:33.583629 7198 raft_consensus.cc:2229] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:33.583638 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ec591f6a444b79a987f6164e4acf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:33 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:36828
I20260929 16:20:33.583739 7199 raft_consensus.cc:2258] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.583818 7200 tablet_replica.cc:330] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: stopping tablet replica
I20260929 16:20:33.583930 7200 raft_consensus.cc:2229] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:33.583948 7198 raft_consensus.cc:2258] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.584064 7200 raft_consensus.cc:2258] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:33.584748 7199 ts_tablet_manager.cc:1905] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.584811 7198 ts_tablet_manager.cc:1905] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.584825 7200 ts_tablet_manager.cc:1905] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:33.587277 7199 ts_tablet_manager.cc:1918] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:33.587277 7200 ts_tablet_manager.cc:1918] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:33.587347 7199 log.cc:1196] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/e0ec591f6a444b79a987f6164e4acf27
I20260929 16:20:33.587359 7200 log.cc:1196] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/e0ec591f6a444b79a987f6164e4acf27
I20260929 16:20:33.587409 7198 ts_tablet_manager.cc:1918] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:20:33.587479 7198 log.cc:1196] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/e0ec591f6a444b79a987f6164e4acf27
I20260929 16:20:33.587613 7199 ts_tablet_manager.cc:1939] T e0ec591f6a444b79a987f6164e4acf27 P f808264d9be546ae8ce4e9f5526d80c1: Deleting consensus metadata
I20260929 16:20:33.587613 7200 ts_tablet_manager.cc:1939] T e0ec591f6a444b79a987f6164e4acf27 P a116ca9e667848e7a7dc6c340183208a: Deleting consensus metadata
I20260929 16:20:33.587734 7198 ts_tablet_manager.cc:1939] T e0ec591f6a444b79a987f6164e4acf27 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting consensus metadata
I20260929 16:20:33.588335 6608 catalog_manager.cc:4841] TS f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991): tablet e0ec591f6a444b79a987f6164e4acf27 (table test-table [id=311a76130c0f415fab2663c74d67497f]) successfully deleted
I20260929 16:20:33.588382 6610 catalog_manager.cc:4841] TS c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607): tablet e0ec591f6a444b79a987f6164e4acf27 (table test-table [id=311a76130c0f415fab2663c74d67497f]) successfully deleted
I20260929 16:20:33.588877 6611 catalog_manager.cc:4841] TS a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039): tablet e0ec591f6a444b79a987f6164e4acf27 (table test-table [id=311a76130c0f415fab2663c74d67497f]) successfully deleted
I20260929 16:20:33.589038 6624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:33.589478 6624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:33.594128 6761 tablet_service.cc:1467] Processing CreateTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 (DEFAULT_TABLE table=test-table [id=9028d4bbca254699818cedd52eb43541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.594266 6901 tablet_service.cc:1467] Processing CreateTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 (DEFAULT_TABLE table=test-table [id=9028d4bbca254699818cedd52eb43541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.594409 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87711ac8548744fcaf4926ef8cc9a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.594543 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87711ac8548744fcaf4926ef8cc9a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.596652 7209 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: Bootstrap starting.
I20260929 16:20:33.596709 7211 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Bootstrap starting.
I20260929 16:20:33.596923 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 (DEFAULT_TABLE table=test-table [id=9028d4bbca254699818cedd52eb43541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:33.597162 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87711ac8548744fcaf4926ef8cc9a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:33.597635 7211 tablet_bootstrap.cc:654] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.597635 7209 tablet_bootstrap.cc:654] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.598924 7209 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: No bootstrap required, opened a new log
I20260929 16:20:33.599006 7209 ts_tablet_manager.cc:1397] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:20:33.599282 7211 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: No bootstrap required, opened a new log
I20260929 16:20:33.599359 7211 ts_tablet_manager.cc:1397] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:33.599653 7209 raft_consensus.cc:348] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.599781 7209 raft_consensus.cc:374] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.599788 7210 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Bootstrap starting.
I20260929 16:20:33.599726 7211 raft_consensus.cc:348] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.599813 7209 raft_consensus.cc:729] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.599857 7211 raft_consensus.cc:374] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.599890 7211 raft_consensus.cc:729] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcba06e9d144b98a36b1c1cf422ee5, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.599944 7209 consensus_queue.cc:260] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.599990 7211 consensus_queue.cc:260] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.600211 7209 ts_tablet_manager.cc:1428] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:20:33.600211 7211 ts_tablet_manager.cc:1428] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:33.600593 7210 tablet_bootstrap.cc:654] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:33.601677 7210 tablet_bootstrap.cc:492] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: No bootstrap required, opened a new log
I20260929 16:20:33.601765 7210 ts_tablet_manager.cc:1397] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:33.602142 7210 raft_consensus.cc:348] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.602253 7210 raft_consensus.cc:374] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:33.602291 7210 raft_consensus.cc:729] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a116ca9e667848e7a7dc6c340183208a, State: Initialized, Role: FOLLOWER
I20260929 16:20:33.602437 7210 consensus_queue.cc:260] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.602721 7210 ts_tablet_manager.cc:1428] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:33.742111 7216 raft_consensus.cc:482] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:33.742264 7216 raft_consensus.cc:504] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.743497 7216 leader_election.cc:288] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607), a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039)
I20260929 16:20:33.749405 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.743934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:49153 (local address 127.4.200.66:35607)
0929 16:20:33.744093 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.744098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.744328 (+ 230us) server_negotiation.cc:407] Connection header received
0929 16:20:33.744426 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.744429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.744474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.744550 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:33.744858 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.745673 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.747992 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.748608 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.748741 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.748792 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.749192 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.749269 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:33.749428 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.743927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:37981 (local address 127.4.200.67:43039)
0929 16:20:33.744087 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.744091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.744281 (+ 190us) server_negotiation.cc:407] Connection header received
0929 16:20:33.744661 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.744664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.744700 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.744754 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:33.745069 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.745779 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.747992 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.748615 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.748740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.748792 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.749192 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.749267 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:33.749593 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.743819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35607 (local address 127.4.200.65:49153)
0929 16:20:33.744254 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.744268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.744344 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.744579 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.744582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.744597 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:33.744710 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.744715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.745802 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.745804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.747833 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.747845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.748757 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.748761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.749025 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.749080 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.749162 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.749290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":283,"thread_start_us":123,"threads_started":1}
I20260929 16:20:33.749742 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.743819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.4.200.65:37981)
0929 16:20:33.744173 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.744187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.744330 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.744798 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.744804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.744825 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:33.744920 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.744924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.745890 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.745895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.747836 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.747845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.748757 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.748761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.749008 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.749083 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.749162 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.749290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":242,"thread_start_us":70,"threads_started":1}
I20260929 16:20:33.750064 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87711ac8548744fcaf4926ef8cc9a433" candidate_uuid: "f808264d9be546ae8ce4e9f5526d80c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" is_pre_election: true
I20260929 16:20:33.750145 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87711ac8548744fcaf4926ef8cc9a433" candidate_uuid: "f808264d9be546ae8ce4e9f5526d80c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a116ca9e667848e7a7dc6c340183208a" is_pre_election: true
I20260929 16:20:33.750224 6921 raft_consensus.cc:2454] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f808264d9be546ae8ce4e9f5526d80c1 in term 0.
I20260929 16:20:33.750313 7061 raft_consensus.cc:2454] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f808264d9be546ae8ce4e9f5526d80c1 in term 0.
I20260929 16:20:33.750670 6714 leader_election.cc:302] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8dcba06e9d144b98a36b1c1cf422ee5, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:33.750979 7216 raft_consensus.cc:2781] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:33.751039 7216 raft_consensus.cc:482] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:33.751077 7216 raft_consensus.cc:3037] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.752079 7216 raft_consensus.cc:504] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.752437 7216 leader_election.cc:288] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [CANDIDATE]: Term 1 election: Requested vote from peers c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607), a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039)
I20260929 16:20:33.752812 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87711ac8548744fcaf4926ef8cc9a433" candidate_uuid: "f808264d9be546ae8ce4e9f5526d80c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5"
I20260929 16:20:33.752856 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87711ac8548744fcaf4926ef8cc9a433" candidate_uuid: "f808264d9be546ae8ce4e9f5526d80c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a116ca9e667848e7a7dc6c340183208a"
I20260929 16:20:33.752947 6921 raft_consensus.cc:3037] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.752956 7061 raft_consensus.cc:3037] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:33.754025 7061 raft_consensus.cc:2454] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f808264d9be546ae8ce4e9f5526d80c1 in term 1.
I20260929 16:20:33.754025 6921 raft_consensus.cc:2454] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f808264d9be546ae8ce4e9f5526d80c1 in term 1.
I20260929 16:20:33.754336 6714 leader_election.cc:302] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8dcba06e9d144b98a36b1c1cf422ee5, f808264d9be546ae8ce4e9f5526d80c1; no voters:
I20260929 16:20:33.754567 7216 raft_consensus.cc:2781] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:33.754843 7216 raft_consensus.cc:686] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 LEADER]: Becoming Leader. State: Replica: f808264d9be546ae8ce4e9f5526d80c1, State: Running, Role: LEADER
I20260929 16:20:33.755122 7216 consensus_queue.cc:237] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } }
I20260929 16:20:33.757675 6623 catalog_manager.cc:5495] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "f808264d9be546ae8ce4e9f5526d80c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f808264d9be546ae8ce4e9f5526d80c1" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 } health_report { overall_health: UNKNOWN } } }
I20260929 16:20:33.805078 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.801542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.4.200.65:37991)
0929 16:20:33.801664 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.801667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.801679 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:33.801811 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.801813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.801863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.801957 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.802134 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.802737 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.803578 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.803787 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.803861 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.804114 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.804646 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.804735 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.804829 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.804877 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:20:33.815315 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.811918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43039 (local address 127.4.200.65:56599)
0929 16:20:33.812200 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.812216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.812314 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.812598 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.812601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.812609 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:33.812737 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.812743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.813444 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.813450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.814079 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.814087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.814575 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.814579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.814674 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.814725 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.815029 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.815036 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.815096 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.815172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260929 16:20:33.815301 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.812135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56599 (local address 127.4.200.67:43039)
0929 16:20:33.812256 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.812261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.812276 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:20:33.812456 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.812463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.812506 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.812588 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.812844 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.813340 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.814199 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.814470 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.814576 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.814871 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.814939 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.815014 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.815106 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.815156 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 16:20:33.821127 6921 raft_consensus.cc:1261] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Refusing update from remote peer f808264d9be546ae8ce4e9f5526d80c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:33.821146 7061 raft_consensus.cc:1261] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Refusing update from remote peer f808264d9be546ae8ce4e9f5526d80c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:33.821624 7240 consensus_queue.cc:1035] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcba06e9d144b98a36b1c1cf422ee5" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:33.821861 7216 consensus_queue.cc:1035] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a116ca9e667848e7a7dc6c340183208a" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:33.839473 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.835538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:37991 (local address 127.4.200.67:39121)
0929 16:20:33.835727 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:33.835742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:33.835822 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:33.836099 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:33.836102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:33.836109 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:20:33.836214 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.836224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.837245 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.837249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.837894 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:33.837904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.838788 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.838792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:33.838879 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.838920 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.839202 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.839211 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:20:33.839275 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:20:33.839357 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 16:20:33.839473 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:33.835580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:39121 (local address 127.4.200.65:37991)
0929 16:20:33.835680 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:33.835686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:33.835925 (+ 239us) server_negotiation.cc:407] Connection header received
0929 16:20:33.835963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:33.835965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:33.836002 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:33.836080 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:33.836327 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.837139 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.838032 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:33.838328 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:33.838781 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:33.839043 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:33.839127 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:33.839200 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:33.839284 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:20:33.839339 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 16:20:33.840291 7182 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:20:35.398597 6624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:35.398778 6624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:35.401371 6624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P accc93c2d5fe47de991687bee8f2e36d: Sending DeleteTablet for 3 replicas of tablet 87711ac8548744fcaf4926ef8cc9a433
I20260929 16:20:35.401975 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:35 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260929 16:20:35.402067 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:35 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:36828
I20260929 16:20:35.402141 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet 87711ac8548744fcaf4926ef8cc9a433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:35 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:38382
I20260929 16:20:35.402338 7271 tablet_replica.cc:330] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1: stopping tablet replica
I20260929 16:20:35.402469 7272 tablet_replica.cc:330] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: stopping tablet replica
I20260929 16:20:35.402505 7271 raft_consensus.cc:2229] T 87711ac8548744fcaf4926ef8cc9a433 P f808264d9be546ae8ce4e9f5526d80c1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:35.402606 7272 raft_consensus.cc:2229] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:35.402734 7272 raft_consensus.cc:2258] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:35.402993 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6696
I20260929 16:20:35.404440 7272 ts_tablet_manager.cc:1905] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:35.406834 7272 ts_tablet_manager.cc:1918] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:20:35.406901 7272 log.cc:1196] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/87711ac8548744fcaf4926ef8cc9a433
I20260929 16:20:35.407166 7272 ts_tablet_manager.cc:1939] T 87711ac8548744fcaf4926ef8cc9a433 P a116ca9e667848e7a7dc6c340183208a: Deleting consensus metadata
I20260929 16:20:35.407136 7273 tablet_replica.cc:330] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: stopping tablet replica
I20260929 16:20:35.407341 7273 raft_consensus.cc:2229] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:35.407496 7273 raft_consensus.cc:2258] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:35.408041 6611 catalog_manager.cc:4841] TS a116ca9e667848e7a7dc6c340183208a (127.4.200.67:43039): tablet 87711ac8548744fcaf4926ef8cc9a433 (table test-table [id=9028d4bbca254699818cedd52eb43541]) successfully deleted
I20260929 16:20:35.408255 7273 ts_tablet_manager.cc:1905] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:35.411872 7273 ts_tablet_manager.cc:1918] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:20:35.411974 7273 log.cc:1196] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/87711ac8548744fcaf4926ef8cc9a433
I20260929 16:20:35.412276 7273 ts_tablet_manager.cc:1939] T 87711ac8548744fcaf4926ef8cc9a433 P c8dcba06e9d144b98a36b1c1cf422ee5: Deleting consensus metadata
I20260929 16:20:35.413201 6610 catalog_manager.cc:4841] TS c8dcba06e9d144b98a36b1c1cf422ee5 (127.4.200.66:35607): tablet 87711ac8548744fcaf4926ef8cc9a433 (table test-table [id=9028d4bbca254699818cedd52eb43541]) successfully deleted
W20260929 16:20:35.414141 6608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:37991 (error 108)
I20260929 16:20:35.414428 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6836
I20260929 16:20:35.415511 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:35.414996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:37991 (local address 127.0.0.1:55968)
0929 16:20:35.415286 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:35.415382 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:37991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
W20260929 16:20:35.415761 6608 catalog_manager.cc:4584] TS f808264d9be546ae8ce4e9f5526d80c1 (127.4.200.65:37991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87711ac8548744fcaf4926ef8cc9a433: Network error: Client connection negotiation failed: client connection to 127.4.200.65:37991: connect: Connection refused (error 111)
I20260929 16:20:35.423046 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6976
I20260929 16:20:35.430787 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 6592
2026-09-29T16:20:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6469 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:35 dist-test-slave-s9ct krb5kdc[7278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:35 dist-test-slave-s9ct krb5kdc[7278](info): set up 2 sockets
Sep 29 16:20:35 dist-test-slave-s9ct krb5kdc[7278](info): commencing operation
krb5kdc: starting...
W20260929 16:20:36.486481 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:36 dist-test-slave-s9ct krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:36.613473 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36921
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:36.723130 7291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:36.723439 7291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:36.723521 7291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:36.727660 7291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:36.727756 7291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:36.727793 7291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:36.727854 7291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:36.727916 7291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:36.733336 7291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:36.734579 7291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:36.735652 7291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:36.744783 7291 server_base.cc:1034] running on GCE node
W20260929 16:20:36.744830 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:36.744844 7296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:36.744961 7299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:36.745639 7291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:36.746954 7291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:36.748117 7291 hybrid_clock.cc:648] HybridClock initialized: now 1790698836748092 us; error 48 us; skew 500 ppm
Sep 29 16:20:36 dist-test-slave-s9ct krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698836, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:36.751091 7291 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:36.751979 7291 webserver.cc:460] Webserver started at http://127.4.200.126:33255/ using document root <none> and password file <none>
I20260929 16:20:36.752557 7291 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:36.752651 7291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:36.752887 7291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:36.755105 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "98b734e31e864f3bb25bc411666b68d4"
format_stamp: "Formatted at 2026-09-29 16:20:36 on dist-test-slave-s9ct"
server_key: "68b16450f9bb8f2eefb206070cedbcff"
server_key_iv: "266392d35fd0d708a60461a316ac873c"
server_key_version: "encryptionkey@0"
I20260929 16:20:36.755671 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "98b734e31e864f3bb25bc411666b68d4"
format_stamp: "Formatted at 2026-09-29 16:20:36 on dist-test-slave-s9ct"
server_key: "68b16450f9bb8f2eefb206070cedbcff"
server_key_iv: "266392d35fd0d708a60461a316ac873c"
server_key_version: "encryptionkey@0"
I20260929 16:20:36.759454 7291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:36.761878 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:36.763149 7291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:36.763291 7291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "98b734e31e864f3bb25bc411666b68d4"
format_stamp: "Formatted at 2026-09-29 16:20:36 on dist-test-slave-s9ct"
server_key: "68b16450f9bb8f2eefb206070cedbcff"
server_key_iv: "266392d35fd0d708a60461a316ac873c"
server_key_version: "encryptionkey@0"
I20260929 16:20:36.763405 7291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:36.773156 7291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:36.777208 7291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:36.777441 7291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:36.784966 7291 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36921
I20260929 16:20:36.785032 7358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36921 every 8 connection(s)
I20260929 16:20:36.785954 7291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:36.788745 7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:36.789391 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7291
I20260929 16:20:36.789549 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:36.789834 4897 external_mini_cluster.cc:1428] Setting key 429b4e7ad391a504c5982c2d26c796d5
I20260929 16:20:36.795413 7359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: Bootstrap starting.
I20260929 16:20:36.797904 7359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:36.798712 7359 log.cc:824] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:20:36 dist-test-slave-s9ct krb5kdc[7278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:36.800954 7359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: No bootstrap required, opened a new log
I20260929 16:20:36.804464 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:36.791333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.4.200.126:36921)
0929 16:20:36.791752 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:36.791771 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:36.791816 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:20:36.792627 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:36.792649 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:36.793056 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:36.793415 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:36.794075 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:36.795564 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:36.796457 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:36.797142 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:36.797300 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:36.799898 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:36.799903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:36.799916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:36.799943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:36.801653 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:36.802141 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:36.802146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:36.802150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:36.802254 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:36.802566 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:36.802568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:36.802569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:36.802947 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:36.803097 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:36.803404 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:20:36.803723 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260929 16:20:36.804989 7359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } }
I20260929 16:20:36.805235 7359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:36.805317 7359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b734e31e864f3bb25bc411666b68d4, State: Initialized, Role: FOLLOWER
I20260929 16:20:36.805811 7359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } }
I20260929 16:20:36.805966 7359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:36.806061 7359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:36.806176 7359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:36.807209 7359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } }
I20260929 16:20:36.807554 7359 leader_election.cc:302] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98b734e31e864f3bb25bc411666b68d4; no voters:
I20260929 16:20:36.807909 7359 leader_election.cc:288] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:36.808053 7364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:36.808324 7364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [term 1 LEADER]: Becoming Leader. State: Replica: 98b734e31e864f3bb25bc411666b68d4, State: Running, Role: LEADER
I20260929 16:20:36.808681 7364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } }
I20260929 16:20:36.809154 7359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:36.810326 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98b734e31e864f3bb25bc411666b68d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } } }
I20260929 16:20:36.810480 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:36.810571 7366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b734e31e864f3bb25bc411666b68d4. Latest consensus state: current_term: 1 leader_uuid: "98b734e31e864f3bb25bc411666b68d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b734e31e864f3bb25bc411666b68d4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36921 } } }
I20260929 16:20:36.810669 7366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4 [sys.catalog]: This master's current role is: LEADER
W20260929 16:20:36.814239 7379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:36.814601 7379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:36.814704 7373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:36.815649 7373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:36.821583 7373 catalog_manager.cc:1347] Generated new cluster ID: 15da0b2076f548c89c097f17702cfcc7
I20260929 16:20:36.821689 7373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:36.837270 7373 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:36.838515 7373 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:36.850805 7373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98b734e31e864f3bb25bc411666b68d4: Generated new TSK 0
I20260929 16:20:36.851585 7373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:36.889516 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:37.003060 7385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:37.003350 7385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:37.003394 7385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:37.007704 7385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:37.007771 7385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:37.007861 7385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:37.013593 7385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:37.014744 7385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:37.015806 7385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:37.025718 7390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.025733 7391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.025738 7393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:37.026173 7385 server_base.cc:1034] running on GCE node
I20260929 16:20:37.026607 7385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:37.027194 7385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:37.028352 7385 hybrid_clock.cc:648] HybridClock initialized: now 1790698837028331 us; error 36 us; skew 500 ppm
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:37.031229 7385 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:37.032101 7385 webserver.cc:460] Webserver started at http://127.4.200.65:36893/ using document root <none> and password file <none>
I20260929 16:20:37.032613 7385 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:37.032670 7385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:37.032830 7385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:37.034860 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "500c2e6cc31b477c8d4d351d6d239780"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "aa156090068a8a4fe95a1eab48b46648"
server_key_iv: "82bf737286fa2f276c98027752d6583c"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.035357 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "500c2e6cc31b477c8d4d351d6d239780"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "aa156090068a8a4fe95a1eab48b46648"
server_key_iv: "82bf737286fa2f276c98027752d6583c"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.038846 7385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:37.041085 7400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.042239 7385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:37.042368 7385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "500c2e6cc31b477c8d4d351d6d239780"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "aa156090068a8a4fe95a1eab48b46648"
server_key_iv: "82bf737286fa2f276c98027752d6583c"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.042531 7385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:37.076050 7385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:37.079522 7385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:37.079757 7385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:37.080384 7385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:37.081262 7385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:37.081336 7385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.081408 7385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:37.081460 7385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.091909 7385 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:45205
I20260929 16:20:37.091938 7513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:45205 every 8 connection(s)
I20260929 16:20:37.092836 7385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:37.096774 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7385
I20260929 16:20:37.096897 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:37.097183 4897 external_mini_cluster.cc:1428] Setting key 803f4aba2ca0a065c3703481629e4c62
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:37.109095 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.094767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:54795 (local address 127.4.200.126:36921)
0929 16:20:37.094914 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:37.094919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:37.095809 (+ 890us) server_negotiation.cc:407] Connection header received
0929 16:20:37.097218 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:37.097223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:37.097282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:37.097379 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:37.098814 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.099601 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.100922 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.101322 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.101467 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.104759 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:37.104763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:37.104765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:37.104803 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:37.106660 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.107338 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.107342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.107344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.107433 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.107811 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.107815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.107816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.108033 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:37.108135 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:37.108829 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 16:20:37.108965 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:20:37.110298 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.095212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36921 (local address 127.4.200.65:54795)
0929 16:20:37.095658 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:37.095699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:37.096992 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:37.097578 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:37.097585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:37.098031 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:37.098689 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.098704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.099765 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.099769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.100789 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.100799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.101520 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.101523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.101683 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.102336 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:37.102353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:37.104352 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:37.106800 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.106807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.106827 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.107215 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.107548 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.107550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.107552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.107694 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.108169 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:37.108175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:37.108539 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 16:20:37.108755 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 16:20:37.109040 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260929 16:20:37.111688 7514 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36921
I20260929 16:20:37.111986 7514 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:37.112500 7514 heartbeater.cc:511] Master 127.4.200.126:36921 requested a full tablet report, sending...
I20260929 16:20:37.113994 7323 ts_manager.cc:194] Registered new tserver with Master: 500c2e6cc31b477c8d4d351d6d239780 (127.4.200.65:45205)
I20260929 16:20:37.115207 7323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:54795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:37.125886 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:37.235284 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:37.235533 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:37.235575 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:37.239481 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:37.239531 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:37.239586 7519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:37.244649 7519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:37.245622 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:37.246863 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:37.256605 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.256632 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.256605 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:37.256960 7519 server_base.cc:1034] running on GCE node
I20260929 16:20:37.257390 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:37.258013 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:37.259207 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790698837259195 us; error 48 us; skew 500 ppm
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:37.261914 7519 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:37.262898 7519 webserver.cc:460] Webserver started at http://127.4.200.66:40251/ using document root <none> and password file <none>
I20260929 16:20:37.263453 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:37.263542 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:37.263772 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:37.265790 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "4019bf1bcab749e387190d6d82955a9b"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "e13f9ce05dea1255606c15294a6d3dc2"
server_key_iv: "512610329e90891a5b9ef1cbf69f449d"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.266356 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "4019bf1bcab749e387190d6d82955a9b"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "e13f9ce05dea1255606c15294a6d3dc2"
server_key_iv: "512610329e90891a5b9ef1cbf69f449d"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.270138 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:37.272518 7534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.273710 7519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:20:37.273849 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "4019bf1bcab749e387190d6d82955a9b"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "e13f9ce05dea1255606c15294a6d3dc2"
server_key_iv: "512610329e90891a5b9ef1cbf69f449d"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.273970 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:37.286782 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:37.289412 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:37.289608 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:37.290179 7519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:37.291023 7519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:37.291093 7519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.291164 7519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:37.291208 7519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.300109 7519 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:32807
I20260929 16:20:37.300129 7647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:32807 every 8 connection(s)
I20260929 16:20:37.300966 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:37.301800 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7519
I20260929 16:20:37.301934 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:37.302219 4897 external_mini_cluster.cc:1428] Setting key cb15b6ca77c0387f4a463f03604717e8
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:37.316841 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.302741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:53325 (local address 127.4.200.126:36921)
0929 16:20:37.302890 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:37.302895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:37.303947 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 16:20:37.305978 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:37.305981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:37.306032 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:37.306109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:37.307314 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.308204 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.309478 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.309924 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.310063 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.312996 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:37.313002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:37.313007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:37.313042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:37.314687 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.315187 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.315190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.315191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.315267 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.315702 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.315705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.315707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.315890 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:37.315969 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:37.316618 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 16:20:37.316719 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:20:37.317764 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.303218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36921 (local address 127.4.200.66:53325)
0929 16:20:37.303772 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:37.303847 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:37.305740 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:37.306293 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:37.306299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:37.306702 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:37.307175 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.307188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.308356 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.308359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.309350 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.309360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.310104 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.310108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.310379 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.311152 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:37.311171 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:37.312828 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:37.314802 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.314808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.314822 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.315081 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.315439 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.315447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.315450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.315606 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.315997 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:37.316006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:37.316341 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 16:20:37.316564 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:20:37.316920 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":126,"threads_started":1}
I20260929 16:20:37.319002 7648 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36921
I20260929 16:20:37.319231 7648 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:37.319730 7648 heartbeater.cc:511] Master 127.4.200.126:36921 requested a full tablet report, sending...
I20260929 16:20:37.320791 7323 ts_manager.cc:194] Registered new tserver with Master: 4019bf1bcab749e387190d6d82955a9b (127.4.200.66:32807)
I20260929 16:20:37.321365 7323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:53325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:37.333397 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:37.442780 7653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:37.443033 7653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:37.443075 7653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:37.447304 7653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:37.447367 7653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:37.447451 7653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:37.452731 7653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:37.453744 7653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:37.454787 7653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:37.464169 7658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.464219 7659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:37.464187 7661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:37.464342 7653 server_base.cc:1034] running on GCE node
I20260929 16:20:37.464977 7653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:37.465626 7653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:37.466816 7653 hybrid_clock.cc:648] HybridClock initialized: now 1790698837466787 us; error 51 us; skew 500 ppm
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:37.469655 7653 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:37.470597 7653 webserver.cc:460] Webserver started at http://127.4.200.67:45811/ using document root <none> and password file <none>
I20260929 16:20:37.471153 7653 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:37.471240 7653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:37.471469 7653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:37.473481 7653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "924cd69f511545d88e08372b8fe3768f"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "fcd11fa61807a6ceeed12cb4e31987f0"
server_key_iv: "b751f2ba82a35e0c54e6b085e1aa0c16"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.474044 7653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "924cd69f511545d88e08372b8fe3768f"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "fcd11fa61807a6ceeed12cb4e31987f0"
server_key_iv: "b751f2ba82a35e0c54e6b085e1aa0c16"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.477821 7653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:20:37.480408 7668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.481717 7653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:37.481873 7653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "924cd69f511545d88e08372b8fe3768f"
format_stamp: "Formatted at 2026-09-29 16:20:37 on dist-test-slave-s9ct"
server_key: "fcd11fa61807a6ceeed12cb4e31987f0"
server_key_iv: "b751f2ba82a35e0c54e6b085e1aa0c16"
server_key_version: "encryptionkey@0"
I20260929 16:20:37.481997 7653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:37.494560 7653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:37.497786 7653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:37.498003 7653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:37.498679 7653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:37.499526 7653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:37.499600 7653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.499668 7653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:37.499719 7653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:37.509498 7653 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:36835
I20260929 16:20:37.509580 7781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:36835 every 8 connection(s)
I20260929 16:20:37.510650 7653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:37.519954 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7653
I20260929 16:20:37.520088 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:37.520383 4897 external_mini_cluster.cc:1428] Setting key d6fb358c322d8ce4c4fb069ec933adda
I20260929 16:20:37.524667 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.512554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41219 (local address 127.4.200.126:36921)
0929 16:20:37.512699 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:37.512703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:37.513467 (+ 764us) server_negotiation.cc:407] Connection header received
0929 16:20:37.514777 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:37.514780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:37.514828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:37.514901 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:37.516001 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.516538 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.517659 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.517876 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.517960 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.520608 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:37.520612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:37.520614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:37.520638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:37.522549 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.523213 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.523217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.523218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.523299 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:37.523659 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:37.523661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:37.523663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:37.523843 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:37.523930 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:37.524437 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 16:20:37.524546 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:20:37.525537 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:37.512844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36921 (local address 127.4.200.67:41219)
0929 16:20:37.513278 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:37.513321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:37.514570 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:37.515034 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:37.515040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:37.515399 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:37.515871 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.515883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.516664 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.516667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.517536 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:37.517545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:37.518038 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:37.518042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:37.518198 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:37.518894 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:37.518912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:37.520417 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:37.522683 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.522716 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.522735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.523109 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.523409 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:37.523412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:37.523414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:37.523559 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:37.523960 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:37.523966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:37.524215 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 16:20:37.524383 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 16:20:37.524652 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260929 16:20:37.526700 7782 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36921
I20260929 16:20:37.526916 7782 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:37.527371 7782 heartbeater.cc:511] Master 127.4.200.126:36921 requested a full tablet report, sending...
I20260929 16:20:37.528426 7323 ts_manager.cc:194] Registered new tserver with Master: 924cd69f511545d88e08372b8fe3768f (127.4.200.67:36835)
I20260929 16:20:37.529047 7323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:41219
I20260929 16:20:37.534833 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:20:37.548394 7362 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:37.546775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.4.200.126:36921)
0929 16:20:37.546931 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:37.546937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:37.547171 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:20:37.547355 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:37.547358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:37.547402 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:37.547487 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:37.548237 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46846: BlockingRecv error: recv got EOF from 127.0.0.1:46846 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:37.549703 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7385
I20260929 16:20:37.555153 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7519
I20260929 16:20:37.560648 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7653
I20260929 16:20:37.566057 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7291
2026-09-29T16:20:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2133 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7794](info): set up 2 sockets
Sep 29 16:20:37 dist-test-slave-s9ct krb5kdc[7794](info): commencing operation
krb5kdc: starting...
W20260929 16:20:38.606774 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:38 dist-test-slave-s9ct krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:38.730057 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:35029
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:38.842962 7807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:38.843261 7807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:38.843303 7807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:38.847455 7807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:38.847527 7807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:38.847548 7807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:38.847564 7807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:38.847580 7807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:38.852980 7807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:38.854043 7807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:38.855542 7807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:38.868194 7812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:38.868372 7813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:38.868440 7807 server_base.cc:1034] running on GCE node
W20260929 16:20:38.868221 7815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:38.869220 7807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:38.870657 7807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:38.871908 7807 hybrid_clock.cc:648] HybridClock initialized: now 1790698838871871 us; error 50 us; skew 500 ppm
Sep 29 16:20:38 dist-test-slave-s9ct krb5kdc[7794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:38.875542 7807 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:38.877239 7807 webserver.cc:460] Webserver started at http://127.4.200.126:46499/ using document root <none> and password file <none>
I20260929 16:20:38.877874 7807 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:38.877974 7807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:38.878216 7807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:38.881242 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "8a2b2295e8f04c41b6aabeecc43085b0"
format_stamp: "Formatted at 2026-09-29 16:20:38 on dist-test-slave-s9ct"
server_key: "c5f023111f748c2809deffea3849f600"
server_key_iv: "5e7c7bb477c8fc1b82de37c1476441b4"
server_key_version: "encryptionkey@0"
I20260929 16:20:38.881963 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "8a2b2295e8f04c41b6aabeecc43085b0"
format_stamp: "Formatted at 2026-09-29 16:20:38 on dist-test-slave-s9ct"
server_key: "c5f023111f748c2809deffea3849f600"
server_key_iv: "5e7c7bb477c8fc1b82de37c1476441b4"
server_key_version: "encryptionkey@0"
I20260929 16:20:38.887249 7807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:20:38.890908 7822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:38.892735 7807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:20:38.892875 7807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "8a2b2295e8f04c41b6aabeecc43085b0"
format_stamp: "Formatted at 2026-09-29 16:20:38 on dist-test-slave-s9ct"
server_key: "c5f023111f748c2809deffea3849f600"
server_key_iv: "5e7c7bb477c8fc1b82de37c1476441b4"
server_key_version: "encryptionkey@0"
I20260929 16:20:38.893023 7807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:38.930244 7807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:38.933537 7807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:38.933765 7807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:38.941156 7807 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:35029
I20260929 16:20:38.941454 7874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:35029 every 8 connection(s)
I20260929 16:20:38.942174 7807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:38.945127 7875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:38.946772 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7807
I20260929 16:20:38.946872 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:38.947168 4897 external_mini_cluster.cc:1428] Setting key efda093b355ea60223f4d5c01263dc2a
I20260929 16:20:38.951298 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: Bootstrap starting.
I20260929 16:20:38.953591 7875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:38.954555 7875 log.cc:824] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:20:38 dist-test-slave-s9ct krb5kdc[7794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:38.956772 7875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: No bootstrap required, opened a new log
I20260929 16:20:38.961680 7875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } }
I20260929 16:20:38.961956 7875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:38.962029 7875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b2295e8f04c41b6aabeecc43085b0, State: Initialized, Role: FOLLOWER
I20260929 16:20:38.962168 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:38.948566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.4.200.126:35029)
0929 16:20:38.949011 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:38.949026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:38.949062 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:20:38.949719 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:38.949738 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:38.950078 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:38.950443 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:38.951256 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:38.952287 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:38.953224 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:38.953696 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:38.953871 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:38.956800 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:38.956808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:38.956828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:38.956859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:38.958841 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:38.959291 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:38.959300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:38.959307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:38.959478 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:38.959822 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:38.959825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:38.959827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:38.960349 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:38.960559 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:38.960825 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:20:38.961158 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":140,"threads_started":1}
I20260929 16:20:38.962558 7875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } }
I20260929 16:20:38.962718 7875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:38.962782 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:38.962872 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:38.963892 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } }
I20260929 16:20:38.964298 7875 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2b2295e8f04c41b6aabeecc43085b0; no voters:
I20260929 16:20:38.964684 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:38.964812 7880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:38.965046 7880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2b2295e8f04c41b6aabeecc43085b0, State: Running, Role: LEADER
I20260929 16:20:38.965415 7880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } }
I20260929 16:20:38.965478 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:20:38.966049 7875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:38.968255 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2b2295e8f04c41b6aabeecc43085b0. Latest consensus state: current_term: 1 leader_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } } }
I20260929 16:20:38.968375 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:38.968691 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b2295e8f04c41b6aabeecc43085b0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35029 } } }
I20260929 16:20:38.968901 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0 [sys.catalog]: This master's current role is: LEADER
W20260929 16:20:38.971796 7896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:38.971915 7896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:38.972012 7897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:38.972723 7897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:38.979239 7897 catalog_manager.cc:1347] Generated new cluster ID: cd4ba9954c814c8bade78544691ce511
I20260929 16:20:38.979326 7897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:38.985946 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:38.980163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.4.200.126:35029)
0929 16:20:38.980309 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:38.980314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:38.980471 (+ 157us) server_negotiation.cc:407] Connection header received
0929 16:20:38.980619 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:38.980622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:38.980674 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:38.980755 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:38.981397 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:38.982186 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:38.983383 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:38.983750 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:38.983844 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:38.984840 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:38.984844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:38.984847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:38.985460 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:38.985495 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:38.985515 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:38.985597 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:38.985721 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:20:38.985785 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:20:39.006155 7897 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:39.007520 7897 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:39.019227 7897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2b2295e8f04c41b6aabeecc43085b0: Generated new TSK 0
I20260929 16:20:39.020124 7897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:20:39.026518 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7807
2026-09-29T16:20:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1458 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:39 dist-test-slave-s9ct krb5kdc[7908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:39 dist-test-slave-s9ct krb5kdc[7908](info): set up 2 sockets
Sep 29 16:20:39 dist-test-slave-s9ct krb5kdc[7908](info): commencing operation
krb5kdc: starting...
W20260929 16:20:40.070675 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/test-user.keytab.
W20260929 16:20:41.126689 4897 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.923s user 0.000s sys 0.002s
Sep 29 16:20:41 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
W20260929 16:20:42.393002 4897 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.200.126: real 1.244s user 0.000s sys 0.003s
I20260929 16:20:42.393496 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45483
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:42.509857 7921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:42.510286 7921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:42.510353 7921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:42.514514 7921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:42.514575 7921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:42.514599 7921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:42.514617 7921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:42.514632 7921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:42.520248 7921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:42.521323 7921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:42.522374 7921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:42.531183 7927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:42.531219 7926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:42.531272 7921 server_base.cc:1034] running on GCE node
W20260929 16:20:42.531200 7929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:42.531949 7921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:42.533226 7921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:42.534514 7921 hybrid_clock.cc:648] HybridClock initialized: now 1790698842534506 us; error 39 us; skew 500 ppm
Sep 29 16:20:42 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:42.537457 7921 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:42.538488 7921 webserver.cc:460] Webserver started at http://127.4.200.126:37303/ using document root <none> and password file <none>
I20260929 16:20:42.539155 7921 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:42.539256 7921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:42.539526 7921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:42.541767 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff"
format_stamp: "Formatted at 2026-09-29 16:20:42 on dist-test-slave-s9ct"
server_key: "5e45902ceefde6d42aafb15367d4573d"
server_key_iv: "14654040666ac21d86f225f5801afe52"
server_key_version: "encryptionkey@0"
I20260929 16:20:42.542462 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff"
format_stamp: "Formatted at 2026-09-29 16:20:42 on dist-test-slave-s9ct"
server_key: "5e45902ceefde6d42aafb15367d4573d"
server_key_iv: "14654040666ac21d86f225f5801afe52"
server_key_version: "encryptionkey@0"
I20260929 16:20:42.546612 7921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:20:42.549168 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:42.550808 7921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:42.550940 7921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff"
format_stamp: "Formatted at 2026-09-29 16:20:42 on dist-test-slave-s9ct"
server_key: "5e45902ceefde6d42aafb15367d4573d"
server_key_iv: "14654040666ac21d86f225f5801afe52"
server_key_version: "encryptionkey@0"
I20260929 16:20:42.551074 7921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:42.579773 7921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:42.583515 7921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:42.583775 7921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:42.591317 7921 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45483
I20260929 16:20:42.591621 7988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45483 every 8 connection(s)
I20260929 16:20:42.592432 7921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:42.595598 7989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:42.600175 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 7921
I20260929 16:20:42.600332 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:42.600632 4897 external_mini_cluster.cc:1428] Setting key 746fba06c4d7ccfe00859b794dfe7d17
I20260929 16:20:42.601760 7989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff: Bootstrap starting.
I20260929 16:20:42.604609 7989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:42.605477 7989 log.cc:824] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:42.607707 7989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff: No bootstrap required, opened a new log
Sep 29 16:20:42 dist-test-slave-s9ct krb5kdc[7908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:42.611819 7989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } }
I20260929 16:20:42.612140 7989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:42.612200 7989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085e4d0e35304c8d9ee9eaf3deedc1ff, State: Initialized, Role: FOLLOWER
I20260929 16:20:42.612792 7989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } }
I20260929 16:20:42.612957 7989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:42.613027 7989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:42.613126 7989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:42.616441 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:42.602154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.4.200.126:45483)
0929 16:20:42.602640 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:42.602659 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:42.602703 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:20:42.603596 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:42.603620 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:42.604058 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:42.604435 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:42.605295 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:42.606636 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:42.607599 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:42.608284 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:42.608440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:42.611337 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:42.611344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:42.611360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:42.611392 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:42.613537 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:42.614002 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:42.614009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:42.614015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:42.614179 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:42.614469 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:42.614471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:42.614472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:42.614864 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:42.615007 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:42.615344 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:20:42.615652 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":207,"threads_started":1}
I20260929 16:20:42.678099 7989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } }
I20260929 16:20:42.678823 7989 leader_election.cc:302] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 085e4d0e35304c8d9ee9eaf3deedc1ff; no voters:
I20260929 16:20:42.679323 7989 leader_election.cc:288] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:42.679494 7994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:42.679793 7994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [term 1 LEADER]: Becoming Leader. State: Replica: 085e4d0e35304c8d9ee9eaf3deedc1ff, State: Running, Role: LEADER
I20260929 16:20:42.680209 7994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } }
I20260929 16:20:42.680991 7989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:42.681986 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } } }
I20260929 16:20:42.682123 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:42.682464 7996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085e4d0e35304c8d9ee9eaf3deedc1ff. Latest consensus state: current_term: 1 leader_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085e4d0e35304c8d9ee9eaf3deedc1ff" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45483 } } }
I20260929 16:20:42.682651 7996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:42.683574 8001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:42.686458 8001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:42.692013 8001 catalog_manager.cc:1347] Generated new cluster ID: e79215a8c82d440ba4628bb16feca663
I20260929 16:20:42.692080 8001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:42.706966 8001 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:42.707904 8001 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:42.716542 8001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085e4d0e35304c8d9ee9eaf3deedc1ff: Generated new TSK 0
I20260929 16:20:42.717237 8001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
W20260929 16:20:43.988023 4897 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.200.65: real 1.258s user 0.000s sys 0.003s
I20260929 16:20:43.988504 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:44.103222 8015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:44.103538 8015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:44.103646 8015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:44.107815 8015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:44.107920 8015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:44.108018 8015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:44.114966 8015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:44.116336 8015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:44.117416 8015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:44.128563 8020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:44.128669 8015 server_base.cc:1034] running on GCE node
W20260929 16:20:44.128563 8021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:44.128602 8023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:44.129235 8015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:44.129863 8015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:44.131026 8015 hybrid_clock.cc:648] HybridClock initialized: now 1790698844130999 us; error 48 us; skew 500 ppm
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:44.133828 8015 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:44.134807 8015 webserver.cc:460] Webserver started at http://127.4.200.65:39849/ using document root <none> and password file <none>
I20260929 16:20:44.135366 8015 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:44.135424 8015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:44.135656 8015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:44.137655 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "4e921d90117e44dc954c4e5880d06679"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "168bde8a5a80424cc61cc6bb9678425e"
server_key_iv: "874d6238f8a994bab93870a34ec07c48"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.138254 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "4e921d90117e44dc954c4e5880d06679"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "168bde8a5a80424cc61cc6bb9678425e"
server_key_iv: "874d6238f8a994bab93870a34ec07c48"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.141920 8015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:44.144292 8030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.145592 8015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:44.145684 8015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "4e921d90117e44dc954c4e5880d06679"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "168bde8a5a80424cc61cc6bb9678425e"
server_key_iv: "874d6238f8a994bab93870a34ec07c48"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.145810 8015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:44.180778 8015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:44.184273 8015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:44.184473 8015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:44.185075 8015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:44.185891 8015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:44.185936 8015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.186008 8015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:44.186040 8015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.194953 8015 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:43749
I20260929 16:20:44.194969 8143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:43749 every 8 connection(s)
I20260929 16:20:44.195854 8015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:44.205775 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8015
I20260929 16:20:44.205945 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:44.206234 4897 external_mini_cluster.cc:1428] Setting key 3ca1f4a070aa6866ec36ec91bc526874
I20260929 16:20:44.209949 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.197644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:57545 (local address 127.4.200.126:45483)
0929 16:20:44.197903 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:44.197909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:44.198725 (+ 816us) server_negotiation.cc:407] Connection header received
0929 16:20:44.199996 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:44.199999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:44.200046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:44.200126 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:44.201270 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.201850 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.203013 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.203256 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.203352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.205929 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:44.205934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:44.205937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:44.205968 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:44.207894 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.208395 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.208400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.208402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.208486 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.208828 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.208832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.208833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.209014 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:44.209124 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:44.209626 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:20:44.209751 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260929 16:20:44.210790 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.198083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45483 (local address 127.4.200.65:57545)
0929 16:20:44.198586 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:44.198628 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:44.199813 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:44.200275 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:44.200282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:44.200635 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:44.201111 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.201123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.201964 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.201967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.202888 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.202897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.203390 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.203392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.203535 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.204146 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:44.204161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:44.205694 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:44.208011 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.208017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.208039 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.208287 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.208584 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.208587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.208589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.208725 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.209134 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:44.209143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:44.209381 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 16:20:44.209586 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:20:44.209912 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260929 16:20:44.212020 8144 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45483
I20260929 16:20:44.212282 8144 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:44.212915 8144 heartbeater.cc:511] Master 127.4.200.126:45483 requested a full tablet report, sending...
I20260929 16:20:44.214780 7953 ts_manager.cc:194] Registered new tserver with Master: 4e921d90117e44dc954c4e5880d06679 (127.4.200.65:43749)
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:44.216449 7953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:57545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:44.233592 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:44.344273 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:44.344547 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:44.344590 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:44.348664 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:44.348729 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:44.348830 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:44.354275 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:44.355501 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:44.356583 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:44.366976 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:44.367010 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:44.366976 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:44.367246 8150 server_base.cc:1034] running on GCE node
I20260929 16:20:44.367704 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:44.368361 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:44.369565 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790698844369533 us; error 26 us; skew 500 ppm
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:44.372329 8150 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:44.373193 8150 webserver.cc:460] Webserver started at http://127.4.200.66:37433/ using document root <none> and password file <none>
I20260929 16:20:44.373755 8150 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:44.373847 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:44.374079 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:44.376119 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "9506aba28a8945088b49f2d7ffb3a3e4"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "b571e5847cabbd1b2751eda3e125bb18"
server_key_iv: "9f2e562d6a4d4b7c5952ae6fc62288ad"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.376682 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "9506aba28a8945088b49f2d7ffb3a3e4"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "b571e5847cabbd1b2751eda3e125bb18"
server_key_iv: "9f2e562d6a4d4b7c5952ae6fc62288ad"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.380403 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:20:44.383078 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.384394 8150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:44.384534 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "9506aba28a8945088b49f2d7ffb3a3e4"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "b571e5847cabbd1b2751eda3e125bb18"
server_key_iv: "9f2e562d6a4d4b7c5952ae6fc62288ad"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.384670 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:44.408587 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:44.411711 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:44.411916 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:44.412503 8150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:44.413329 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:44.413400 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.413472 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:44.413524 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.422220 8150 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38555
I20260929 16:20:44.422243 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38555 every 8 connection(s)
I20260929 16:20:44.423135 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:44.430694 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8150
I20260929 16:20:44.430822 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:44.431129 4897 external_mini_cluster.cc:1428] Setting key 9f5bcfae568197310d7bc789cb0f9132
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:44.436921 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.424932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:57739 (local address 127.4.200.126:45483)
0929 16:20:44.425107 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:44.425112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:44.425910 (+ 798us) server_negotiation.cc:407] Connection header received
0929 16:20:44.427238 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:44.427241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:44.427287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:44.427361 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:44.428421 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.428964 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.430097 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.430310 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.430445 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.433175 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:44.433182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:44.433188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:44.433219 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:44.434815 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.435419 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.435422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.435423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.435493 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.435870 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.435872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.435874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.436032 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:44.436106 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:44.436705 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 16:20:44.436803 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:20:44.438139 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.425322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45483 (local address 127.4.200.66:57739)
0929 16:20:44.425757 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:44.425798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:44.427051 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:44.427487 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:44.427493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:44.427834 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:44.428299 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.428312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.429085 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.429088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.429985 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.429994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.430485 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.430491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.430716 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.431320 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:44.431335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:44.433009 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:44.434927 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.434935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.434951 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.435322 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.435585 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.435589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.435593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.435778 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.436138 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:44.436148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:44.436454 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 16:20:44.436675 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 16:20:44.437038 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260929 16:20:44.439404 8279 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45483
I20260929 16:20:44.439644 8279 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:44.440110 8279 heartbeater.cc:511] Master 127.4.200.126:45483 requested a full tablet report, sending...
I20260929 16:20:44.441138 7953 ts_manager.cc:194] Registered new tserver with Master: 9506aba28a8945088b49f2d7ffb3a3e4 (127.4.200.66:38555)
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:20:44.441735 7953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:57739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:44.461058 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:44.571731 8284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:44.572018 8284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:44.572062 8284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:44.576295 8284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:44.576352 8284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:44.576412 8284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:44.581790 8284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:44.582926 8284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:44.583953 8284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:44.593335 8289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:44.593327 8290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:44.593331 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:44.593735 8284 server_base.cc:1034] running on GCE node
I20260929 16:20:44.594151 8284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:44.594795 8284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:44.595973 8284 hybrid_clock.cc:648] HybridClock initialized: now 1790698844595941 us; error 49 us; skew 500 ppm
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:44.598644 8284 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:44.599413 8284 webserver.cc:460] Webserver started at http://127.4.200.67:46169/ using document root <none> and password file <none>
I20260929 16:20:44.599907 8284 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:44.599972 8284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:44.600128 8284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:44.601991 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "a82efd7258cf4831bbe5b33690dae75f"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "76a05375f4c30c4e1bb8877f28f2ac55"
server_key_iv: "6f1403c875510fb022701a6cf817cf32"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.602550 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "a82efd7258cf4831bbe5b33690dae75f"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "76a05375f4c30c4e1bb8877f28f2ac55"
server_key_iv: "6f1403c875510fb022701a6cf817cf32"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.605960 8284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:44.608224 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.609359 8284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:44.609472 8284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "a82efd7258cf4831bbe5b33690dae75f"
format_stamp: "Formatted at 2026-09-29 16:20:44 on dist-test-slave-s9ct"
server_key: "76a05375f4c30c4e1bb8877f28f2ac55"
server_key_iv: "6f1403c875510fb022701a6cf817cf32"
server_key_version: "encryptionkey@0"
I20260929 16:20:44.609570 8284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:44.630048 8284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:44.633287 8284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:44.633431 8284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:44.634050 8284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:44.634852 8284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:44.634896 8284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.634968 8284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:44.635010 8284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:44.643270 8284 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:42671
I20260929 16:20:44.643289 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:42671 every 8 connection(s)
I20260929 16:20:44.644114 8284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:44.647104 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8284
I20260929 16:20:44.647230 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:44.647583 4897 external_mini_cluster.cc:1428] Setting key 5c8a795fdee926643192ad5502d8867f
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:44.657755 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.645888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41389 (local address 127.4.200.126:45483)
0929 16:20:44.646026 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:44.646030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:44.646864 (+ 834us) server_negotiation.cc:407] Connection header received
0929 16:20:44.648546 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:44.648549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:44.648605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:44.648677 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:44.649991 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.650594 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.651660 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.651882 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.652013 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.654319 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:44.654322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:44.654325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:44.654358 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:44.655771 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.656269 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.656274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.656276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.656365 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.656727 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.656731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.656733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.656905 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:44.657007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:44.657475 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 16:20:44.657590 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:20:44.658542 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.646218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45483 (local address 127.4.200.67:41389)
0929 16:20:44.646708 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:44.646746 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:44.648318 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:44.648838 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:44.648845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:44.649296 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:44.649833 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.649846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.650718 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.650725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.651540 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:44.651549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.652043 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.652047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:44.652208 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.652782 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:44.652797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:44.654161 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:44.655883 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.655889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.655903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.656166 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.656498 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:44.656501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:44.656503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:44.656631 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:44.657015 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:44.657024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:44.657244 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 16:20:44.657425 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:20:44.657715 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":168,"threads_started":1}
I20260929 16:20:44.659544 8413 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45483
I20260929 16:20:44.659770 8413 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:44.660292 8413 heartbeater.cc:511] Master 127.4.200.126:45483 requested a full tablet report, sending...
I20260929 16:20:44.661309 7952 ts_manager.cc:194] Registered new tserver with Master: a82efd7258cf4831bbe5b33690dae75f (127.4.200.67:42671)
I20260929 16:20:44.661867 7952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:41389
I20260929 16:20:44.673171 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[7908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698844, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:44.697948 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:44.689982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.4.200.126:45483)
0929 16:20:44.690110 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:44.690114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:44.690313 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:20:44.690461 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:44.690464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:44.690516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:44.690590 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:44.691323 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.691848 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.692627 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:44.692830 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:44.692905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:44.695383 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:44.695386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:44.695388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:44.695411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:44.696617 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.696967 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.696970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.696971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.697050 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:44.697271 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:44.697273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:44.697274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:44.697476 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:44.697577 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:44.697741 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:20:44.697830 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:20:44.698350 7952 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52800
I20260929 16:20:44.699290 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8015
I20260929 16:20:44.704840 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8150
I20260929 16:20:44.710045 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8284
I20260929 16:20:44.715265 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 7921
2026-09-29T16:20:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5690 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[8425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[8425](info): set up 2 sockets
Sep 29 16:20:44 dist-test-slave-s9ct krb5kdc[8425](info): commencing operation
krb5kdc: starting...
W20260929 16:20:45.746702 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:45 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:45.869927 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36595
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:45.984014 8438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:45.984359 8438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:45.984438 8438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:45.988543 8438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:45.988644 8438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:45.988679 8438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:45.988718 8438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:45.988749 8438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:45.994215 8438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:45.995406 8438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:45.996475 8438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:46.005257 8446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.005275 8443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.005313 8444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.005416 8438 server_base.cc:1034] running on GCE node
I20260929 16:20:46.006213 8438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:46.007366 8438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:46.008554 8438 hybrid_clock.cc:648] HybridClock initialized: now 1790698846008531 us; error 51 us; skew 500 ppm
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:46.011415 8438 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:46.012280 8438 webserver.cc:460] Webserver started at http://127.4.200.126:40015/ using document root <none> and password file <none>
I20260929 16:20:46.012794 8438 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:46.012851 8438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:46.013019 8438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:46.015108 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "db261a7803524363b068d6030a9005d6"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "644087fff7b98aa44493bfe852bd1867"
server_key_iv: "2fa1daade13b0ea22f1cc5ecff017ba0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.015625 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "db261a7803524363b068d6030a9005d6"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "644087fff7b98aa44493bfe852bd1867"
server_key_iv: "2fa1daade13b0ea22f1cc5ecff017ba0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.019498 8438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:46.022004 8453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.023476 8438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:46.023597 8438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "db261a7803524363b068d6030a9005d6"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "644087fff7b98aa44493bfe852bd1867"
server_key_iv: "2fa1daade13b0ea22f1cc5ecff017ba0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.023697 8438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:46.040841 8438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:46.044219 8438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:46.044387 8438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:46.051309 8438 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36595
I20260929 16:20:46.051414 8505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36595 every 8 connection(s)
I20260929 16:20:46.052309 8438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:46.055270 8506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:46.057099 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8438
I20260929 16:20:46.057199 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:46.057487 4897 external_mini_cluster.cc:1428] Setting key 4e6aadd5dd93a08e6eb995c27897324d
I20260929 16:20:46.061723 8506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6: Bootstrap starting.
I20260929 16:20:46.064019 8506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:46.064780 8506 log.cc:824] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:46.066799 8506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6: No bootstrap required, opened a new log
I20260929 16:20:46.070736 8506 raft_consensus.cc:348] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } }
I20260929 16:20:46.070968 8506 raft_consensus.cc:374] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:46.071087 8506 raft_consensus.cc:729] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db261a7803524363b068d6030a9005d6, State: Initialized, Role: FOLLOWER
I20260929 16:20:46.071681 8506 consensus_queue.cc:260] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } }
I20260929 16:20:46.071849 8506 raft_consensus.cc:388] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:46.071931 8506 raft_consensus.cc:482] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:46.072027 8506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:46.072551 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.058914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.4.200.126:36595)
0929 16:20:46.059438 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.059456 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.059494 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:20:46.060163 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.060180 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.060529 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.060904 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.061811 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.062929 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.063818 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.064395 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.064579 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.067395 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.067400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.067415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.067445 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.069264 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.069760 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.069765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.069770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.069886 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.070183 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.070188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.070189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.070668 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.070887 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.071193 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.071506 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":164,"threads_started":1}
I20260929 16:20:46.073222 8506 raft_consensus.cc:504] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } }
I20260929 16:20:46.073608 8506 leader_election.cc:302] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db261a7803524363b068d6030a9005d6; no voters:
I20260929 16:20:46.074011 8506 leader_election.cc:288] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:46.074097 8511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:46.074288 8511 raft_consensus.cc:686] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [term 1 LEADER]: Becoming Leader. State: Replica: db261a7803524363b068d6030a9005d6, State: Running, Role: LEADER
I20260929 16:20:46.074668 8511 consensus_queue.cc:237] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } }
I20260929 16:20:46.075346 8506 sys_catalog.cc:564] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:46.076570 8513 sys_catalog.cc:455] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db261a7803524363b068d6030a9005d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } } }
I20260929 16:20:46.076699 8513 sys_catalog.cc:458] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:46.076822 8512 sys_catalog.cc:455] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db261a7803524363b068d6030a9005d6. Latest consensus state: current_term: 1 leader_uuid: "db261a7803524363b068d6030a9005d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db261a7803524363b068d6030a9005d6" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36595 } } }
I20260929 16:20:46.077026 8512 sys_catalog.cc:458] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:46.077795 8521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:46.080487 8521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:46.086141 8521 catalog_manager.cc:1347] Generated new cluster ID: ddb355b27f0447c08fe106e6a347cd59
I20260929 16:20:46.086212 8521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:46.097273 8521 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:46.098244 8521 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:46.103955 8521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db261a7803524363b068d6030a9005d6: Generated new TSK 0
I20260929 16:20:46.104880 8521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:46.145635 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:46.261770 8532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:46.262064 8532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:46.262110 8532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:46.266287 8532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:46.266364 8532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:46.266446 8532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:46.272204 8532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:46.273466 8532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:46.275055 8532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:46.286150 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.286130 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.286234 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.286319 8532 server_base.cc:1034] running on GCE node
I20260929 16:20:46.286978 8532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:46.287679 8532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:46.288928 8532 hybrid_clock.cc:648] HybridClock initialized: now 1790698846288912 us; error 33 us; skew 500 ppm
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:46.292091 8532 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:46.293098 8532 webserver.cc:460] Webserver started at http://127.4.200.65:46173/ using document root <none> and password file <none>
I20260929 16:20:46.293664 8532 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:46.293731 8532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:46.293920 8532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:46.296197 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "7c17251f0ae844ec9846b1280449bec5"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "f4b24b8198157e506684fda0bbea16d7"
server_key_iv: "c947cab7e479f161d98f6eead0c269d0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.296942 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "7c17251f0ae844ec9846b1280449bec5"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "f4b24b8198157e506684fda0bbea16d7"
server_key_iv: "c947cab7e479f161d98f6eead0c269d0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.301038 8532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:20:46.303985 8547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.305528 8532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:20:46.305686 8532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "7c17251f0ae844ec9846b1280449bec5"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "f4b24b8198157e506684fda0bbea16d7"
server_key_iv: "c947cab7e479f161d98f6eead0c269d0"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.305817 8532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:46.339964 8532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:46.343717 8532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:46.343942 8532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:46.344586 8532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:46.345435 8532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:46.345489 8532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.345558 8532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:46.345708 8532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.356410 8532 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:39499
I20260929 16:20:46.356422 8660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:39499 every 8 connection(s)
I20260929 16:20:46.357465 8532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:46.363035 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8532
I20260929 16:20:46.363145 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:46.363436 4897 external_mini_cluster.cc:1428] Setting key de9861abb23f547a4caed78a91c03cfd
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:46.372761 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.359452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:45229 (local address 127.4.200.126:36595)
0929 16:20:46.359633 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.359639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.360610 (+ 971us) server_negotiation.cc:407] Connection header received
0929 16:20:46.362005 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.362008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.362062 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.362152 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.363539 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.364218 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.365668 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.366093 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.366219 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.368959 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.368963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.368968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.369005 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.370578 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.371131 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.371135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.371137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.371226 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.371595 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.371601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.371603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.371805 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.371916 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.372444 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.372590 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:46.373687 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.359918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36595 (local address 127.4.200.65:45229)
0929 16:20:46.360440 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:46.360486 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:46.361769 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:46.362322 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:46.362329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:46.362806 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:46.363371 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.363395 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.364348 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.364352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.365527 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.365536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.366263 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.366266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.366513 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.367222 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:46.367238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:46.368776 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:46.370705 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.370712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.370730 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.371019 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.371330 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.371333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.371335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.371494 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.371923 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:46.371932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:46.372203 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 16:20:46.372404 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 16:20:46.372762 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":125,"threads_started":1}
I20260929 16:20:46.375204 8661 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36595
I20260929 16:20:46.375514 8661 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:20:46.376236 8661 heartbeater.cc:511] Master 127.4.200.126:36595 requested a full tablet report, sending...
I20260929 16:20:46.378227 8470 ts_manager.cc:194] Registered new tserver with Master: 7c17251f0ae844ec9846b1280449bec5 (127.4.200.65:39499)
I20260929 16:20:46.379454 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:45229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:46.393512 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:46.510469 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:46.510897 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:46.510984 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:46.515228 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:46.515290 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:46.515393 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:46.520964 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:46.522034 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:46.523125 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:46.533282 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.533253 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.533339 8666 server_base.cc:1034] running on GCE node
W20260929 16:20:46.533257 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.533953 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:46.534754 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:46.535918 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790698846535906 us; error 27 us; skew 500 ppm
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:46.546533 8666 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:46.547598 8666 webserver.cc:460] Webserver started at http://127.4.200.66:36793/ using document root <none> and password file <none>
I20260929 16:20:46.548401 8666 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:46.548481 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:46.548720 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:46.551086 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "5c4e96d8c4ee448ea13aada691e46d4d"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "22025d5493ce4778d96520e1801e8259"
server_key_iv: "55978d64bd907ba0e484a39b9b8ffa80"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.551735 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "5c4e96d8c4ee448ea13aada691e46d4d"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "22025d5493ce4778d96520e1801e8259"
server_key_iv: "55978d64bd907ba0e484a39b9b8ffa80"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.555827 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:46.558663 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.560276 8666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:46.560431 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "5c4e96d8c4ee448ea13aada691e46d4d"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "22025d5493ce4778d96520e1801e8259"
server_key_iv: "55978d64bd907ba0e484a39b9b8ffa80"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.560563 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:46.572122 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:46.575440 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:46.575711 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:46.576357 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:46.577343 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:46.577416 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.577483 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:46.577535 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.586517 8666 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:45139
I20260929 16:20:46.586611 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:45139 every 8 connection(s)
I20260929 16:20:46.587478 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:46.590569 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8666
I20260929 16:20:46.590718 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:46.591008 4897 external_mini_cluster.cc:1428] Setting key 0828777eb9e46d52f34f0acbaa34a873
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:20:46.601780 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.589337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:46437 (local address 127.4.200.126:36595)
0929 16:20:46.589535 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.589539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.590332 (+ 793us) server_negotiation.cc:407] Connection header received
0929 16:20:46.592025 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.592028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.592083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.592157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.593323 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.593855 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.595086 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.595320 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.595434 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.597934 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.597943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.597946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.597973 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.599514 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.600128 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.600131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.600133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.600210 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.600554 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.600559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.600560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.600829 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.600941 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.601475 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.601606 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:46.602792 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.589733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36595 (local address 127.4.200.66:46437)
0929 16:20:46.590176 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:46.590216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:46.591802 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:46.592325 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:46.592332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:46.592689 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:46.593170 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.593184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.593989 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.593993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.594956 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.594965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.595480 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.595484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.595663 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.596288 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:46.596306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:46.597743 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:46.599650 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.599656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.599674 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.600011 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.600303 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.600307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.600310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.600455 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.600936 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:46.600954 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:46.601217 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 16:20:46.601412 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:20:46.601755 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260929 16:20:46.604007 8795 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36595
I20260929 16:20:46.604264 8795 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:46.604722 8795 heartbeater.cc:511] Master 127.4.200.126:36595 requested a full tablet report, sending...
I20260929 16:20:46.605751 8470 ts_manager.cc:194] Registered new tserver with Master: 5c4e96d8c4ee448ea13aada691e46d4d (127.4.200.66:45139)
I20260929 16:20:46.606329 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:46437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:46.617875 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:46.731693 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:46.732161 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:46.732270 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:46.736680 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:46.736743 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:46.736805 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:46.742128 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:46.743221 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:46.744279 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:46.753933 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.754005 8800 server_base.cc:1034] running on GCE node
W20260929 16:20:46.753954 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:46.753958 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:46.754674 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:46.755321 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:46.756526 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790698846756498 us; error 42 us; skew 500 ppm
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:46.759415 8800 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:46.760335 8800 webserver.cc:460] Webserver started at http://127.4.200.67:44439/ using document root <none> and password file <none>
I20260929 16:20:46.760914 8800 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:46.761013 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:46.761245 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:46.763402 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "3f6d14822eac468ba17fc6a5a094a1db"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "8765ae86c2f1e2c0a55e27a7fe2502a5"
server_key_iv: "adf8ea9c0be3191c6e678c66f27abd4a"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.764144 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "3f6d14822eac468ba17fc6a5a094a1db"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "8765ae86c2f1e2c0a55e27a7fe2502a5"
server_key_iv: "adf8ea9c0be3191c6e678c66f27abd4a"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.768016 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:46.770583 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.771890 8800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:20:46.772029 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "3f6d14822eac468ba17fc6a5a094a1db"
format_stamp: "Formatted at 2026-09-29 16:20:46 on dist-test-slave-s9ct"
server_key: "8765ae86c2f1e2c0a55e27a7fe2502a5"
server_key_iv: "adf8ea9c0be3191c6e678c66f27abd4a"
server_key_version: "encryptionkey@0"
I20260929 16:20:46.772153 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:46.790954 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:46.794083 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:46.794299 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:46.794950 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:46.795804 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:46.795876 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.795948 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:46.795998 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:46.805220 8800 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:35175
I20260929 16:20:46.805295 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:35175 every 8 connection(s)
I20260929 16:20:46.806180 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:46.814985 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8800
I20260929 16:20:46.815105 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:46.815407 4897 external_mini_cluster.cc:1428] Setting key ad4f84ace8dbc8ea8f740d8dd40f288f
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:46.820571 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.808113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:37811 (local address 127.4.200.126:36595)
0929 16:20:46.808270 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.808274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.809143 (+ 869us) server_negotiation.cc:407] Connection header received
0929 16:20:46.810712 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.810715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.810766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.810850 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.812009 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.812560 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.813711 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.813945 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.814037 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.816925 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.816930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.816936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.816977 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.818605 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.819077 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.819080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.819081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.819157 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.819472 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.819473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.819475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.819650 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.819724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.820302 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.820417 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:20:46.821393 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.808491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36595 (local address 127.4.200.67:37811)
0929 16:20:46.808965 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:46.809019 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:46.810485 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:46.811012 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:46.811019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:46.811369 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:46.811863 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.811876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.812664 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.812666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.813590 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:46.813599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.814081 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.814084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:46.814216 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.814944 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:46.814961 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:46.816761 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:46.818715 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.818720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.818734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.818991 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.819233 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:46.819235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:46.819237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:46.819376 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:46.819771 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:46.819777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:46.820042 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 16:20:46.820221 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:20:46.820500 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260929 16:20:46.822860 8929 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36595
I20260929 16:20:46.823124 8929 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:46.823638 8929 heartbeater.cc:511] Master 127.4.200.126:36595 requested a full tablet report, sending...
I20260929 16:20:46.824808 8470 ts_manager.cc:194] Registered new tserver with Master: 3f6d14822eac468ba17fc6a5a094a1db (127.4.200.67:35175)
I20260929 16:20:46.825394 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:37811
I20260929 16:20:46.829885 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:46.857901 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.847543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.4.200.65:39499)
0929 16:20:46.847854 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.847870 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.847899 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:20:46.848023 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.848050 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.848245 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.848407 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.849134 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.849924 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.850855 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.851255 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.851374 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.853931 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.853942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.853956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.853984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.855769 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.856282 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.856288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.856289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.856383 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.856866 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.856869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.856871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.857189 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.857394 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.857588 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.857731 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:46.870318 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:46.861417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.4.200.126:36595)
0929 16:20:46.861541 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:46.861545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:46.861613 (+ 68us) server_negotiation.cc:407] Connection header received
0929 16:20:46.861761 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:46.861764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:46.861814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:46.861898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:46.862636 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.863165 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.864120 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:46.864341 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:46.864438 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:46.867159 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:46.867163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:46.867165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:46.867193 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:46.868494 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.868937 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.868939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.868941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.869034 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:46.869277 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:46.869279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:46.869280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:46.869512 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:46.869622 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:46.870066 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 16:20:46.870188 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:20:46.870795 8470 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44212
I20260929 16:20:46.871483 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8532
I20260929 16:20:46.877892 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8666
I20260929 16:20:46.884284 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8800
I20260929 16:20:46.889627 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8438
2026-09-29T16:20:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2175 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8941](info): set up 2 sockets
Sep 29 16:20:46 dist-test-slave-s9ct krb5kdc[8941](info): commencing operation
krb5kdc: starting...
W20260929 16:20:47.942714 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:48.069092 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:35201
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:35201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:48.184609 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:48.184890 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:48.184934 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:48.189000 8954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:48.189062 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:48.189081 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:48.189098 8954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:48.189113 8954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:48.194355 8954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:35201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:35201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:48.195487 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:48.196511 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:48.204741 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.204736 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.204739 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:48.205001 8954 server_base.cc:1034] running on GCE node
I20260929 16:20:48.205705 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:48.206912 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:48.208112 8954 hybrid_clock.cc:648] HybridClock initialized: now 1790698848208074 us; error 50 us; skew 500 ppm
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:48.211017 8954 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:48.211807 8954 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:48.211907 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:48.212146 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:48.214182 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "9774e70efa3e4f77811f74053e28def1"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "47e76e6c704346ad9e55c7c23b369895"
server_key_iv: "5d85bfa25bb38d21dde52cdd8909143c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.214807 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "9774e70efa3e4f77811f74053e28def1"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "47e76e6c704346ad9e55c7c23b369895"
server_key_iv: "5d85bfa25bb38d21dde52cdd8909143c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.218488 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:48.220872 8968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.222066 8954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:48.222208 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "9774e70efa3e4f77811f74053e28def1"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "47e76e6c704346ad9e55c7c23b369895"
server_key_iv: "5d85bfa25bb38d21dde52cdd8909143c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.222323 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:48.245219 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:48.248483 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:48.248700 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:48.255944 8954 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:35201
I20260929 16:20:48.256345 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:35201 every 8 connection(s)
I20260929 16:20:48.256660 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:48.259454 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:48.264974 9021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: Bootstrap starting.
I20260929 16:20:48.266134 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 8954
I20260929 16:20:48.266286 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:48.266595 4897 external_mini_cluster.cc:1428] Setting key 6dcd44465a696c87b47fede8111cb2bf
I20260929 16:20:48.268040 9021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:48.268816 9021 log.cc:824] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:48.271080 9021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: No bootstrap required, opened a new log
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:48.275569 9021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } }
I20260929 16:20:48.275821 9021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:48.275882 9021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9774e70efa3e4f77811f74053e28def1, State: Initialized, Role: FOLLOWER
I20260929 16:20:48.276345 9021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } }
I20260929 16:20:48.276503 9021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:48.276582 9021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:48.276703 9021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:48.277585 9021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } }
I20260929 16:20:48.277997 9021 leader_election.cc:302] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9774e70efa3e4f77811f74053e28def1; no voters:
I20260929 16:20:48.278363 9021 leader_election.cc:288] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:48.278468 9026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:48.278697 9026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [term 1 LEADER]: Becoming Leader. State: Replica: 9774e70efa3e4f77811f74053e28def1, State: Running, Role: LEADER
I20260929 16:20:48.279048 9026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } }
I20260929 16:20:48.279736 9021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:48.280259 9026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9774e70efa3e4f77811f74053e28def1. Latest consensus state: current_term: 1 leader_uuid: "9774e70efa3e4f77811f74053e28def1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } } }
I20260929 16:20:48.280465 9026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:48.280864 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9774e70efa3e4f77811f74053e28def1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774e70efa3e4f77811f74053e28def1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 35201 } } }
I20260929 16:20:48.280923 9033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:48.281020 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:48.283895 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:48.267765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.4.200.126:35201)
0929 16:20:48.268188 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:48.268204 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:48.268239 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:20:48.268877 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:48.268903 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:48.269362 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:48.269731 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:48.270598 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.271631 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.272815 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.273300 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.273435 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:48.276369 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:48.276374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:48.276388 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:48.276415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:48.278462 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.279132 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.279137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.279162 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.279316 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.279611 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.279615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.279617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.280127 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:48.280332 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:48.282590 (+ 2258us) server_negotiation.cc:299] Negotiation successful
0929 16:20:48.282923 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":141,"threads_started":1}
I20260929 16:20:48.285244 9033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:48.291435 9033 catalog_manager.cc:1347] Generated new cluster ID: c808dd8fb5dd4ad49c1510ba3a307e9e
I20260929 16:20:48.291508 9033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:48.299711 9033 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:48.300922 9033 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:48.321023 9033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: Generated new TSK 0
I20260929 16:20:48.321923 9033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:48.355898 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:48.481357 9047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:48.481658 9047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:48.481741 9047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:48.486060 9047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:48.486140 9047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:48.486244 9047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:48.491840 9047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:48.492975 9047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:48.494066 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:48.504148 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.504169 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.504623 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:48.504599 9047 server_base.cc:1034] running on GCE node
I20260929 16:20:48.505103 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:48.505728 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:48.509021 9047 hybrid_clock.cc:648] HybridClock initialized: now 1790698848508982 us; error 66 us; skew 500 ppm
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:48.512740 9047 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:48.513538 9047 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:48.513603 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:48.513833 9047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:48.516003 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "197e2b1d296842d58a2696922e25a398"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "b97647223ead51be418c5ac0f3a948d5"
server_key_iv: "453fdaba805100882e355ceca1c7264c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.516569 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "197e2b1d296842d58a2696922e25a398"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "b97647223ead51be418c5ac0f3a948d5"
server_key_iv: "453fdaba805100882e355ceca1c7264c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.520332 9047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:20:48.522974 9061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.524322 9047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:48.524488 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "197e2b1d296842d58a2696922e25a398"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "b97647223ead51be418c5ac0f3a948d5"
server_key_iv: "453fdaba805100882e355ceca1c7264c"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.524612 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:48.551323 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:48.554824 9047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:48.555039 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:48.555660 9047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:48.556506 9047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:48.556584 9047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.556656 9047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:48.556708 9047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.565512 9047 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:46735
I20260929 16:20:48.565531 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:46735 every 8 connection(s)
I20260929 16:20:48.566128 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:48.573725 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9047
I20260929 16:20:48.573868 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:48.574199 4897 external_mini_cluster.cc:1428] Setting key 935c6d0814877b946ba670ead98362ff
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:48.580506 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:48.567855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:48191 (local address 127.4.200.126:35201)
0929 16:20:48.568009 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:48.568013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:48.568818 (+ 805us) server_negotiation.cc:407] Connection header received
0929 16:20:48.570159 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:48.570163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:48.570226 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:48.570333 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:48.571521 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.572058 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.573176 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.573450 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.573570 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:48.576455 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:48.576475 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:48.576478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:48.576503 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:48.578025 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.578637 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.578644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.578648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.578747 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.579121 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.579128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.579133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.579390 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:48.579504 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:48.580072 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 16:20:48.580241 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:20:48.581367 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:48.568212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:35201 (local address 127.4.200.65:48191)
0929 16:20:48.568663 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:48.568705 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:48.569938 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:48.570532 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:48.570539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:48.570892 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:48.571369 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:48.571382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.572187 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.572192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:48.573051 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:48.573059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.573621 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.573625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:48.573813 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:48.574553 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:48.574570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:48.576282 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:48.578164 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:48.578170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:48.578187 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:48.578535 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:48.578861 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:48.578864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:48.578865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:48.579012 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:48.579492 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:48.579497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:48.579788 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 16:20:48.579947 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:20:48.580293 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260929 16:20:48.582787 9175 heartbeater.cc:347] Connected to a master server at 127.4.200.126:35201
I20260929 16:20:48.583033 9175 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:48.583489 9175 heartbeater.cc:511] Master 127.4.200.126:35201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:48.585044 8985 ts_manager.cc:194] Registered new tserver with Master: 197e2b1d296842d58a2696922e25a398 (127.4.200.65:46735)
I20260929 16:20:48.586205 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:48191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:48.603937 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:48.716977 9180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:48.717281 9180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:48.717356 9180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:48.721426 9180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:48.721516 9180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:48.721609 9180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:48.727003 9180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:48.728119 9180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:48.729264 9180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:48.738739 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:48.738747 9180 server_base.cc:1034] running on GCE node
W20260929 16:20:48.738770 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.738992 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:48.739428 9180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:48.740031 9180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:48.741216 9180 hybrid_clock.cc:648] HybridClock initialized: now 1790698848741182 us; error 48 us; skew 500 ppm
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:48.749311 9180 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:48.750133 9180 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:48.750224 9180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:48.750523 9180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:48.752604 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "2bd1f99fff0043f69f148f85c09427f4"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f9a90994909abd6768fb4b0d34327eb"
server_key_iv: "bd6af07a6450eada8e15e8321bacdf80"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.753157 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "2bd1f99fff0043f69f148f85c09427f4"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f9a90994909abd6768fb4b0d34327eb"
server_key_iv: "bd6af07a6450eada8e15e8321bacdf80"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.756824 9180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:20:48.759285 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.760495 9180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:48.760625 9180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "2bd1f99fff0043f69f148f85c09427f4"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f9a90994909abd6768fb4b0d34327eb"
server_key_iv: "bd6af07a6450eada8e15e8321bacdf80"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.760737 9180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:48.775058 9180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:48.778055 9180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:48.778264 9180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:48.778926 9180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:48.779763 9180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:48.779834 9180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.779913 9180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:48.779965 9180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.789350 9180 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:39547
I20260929 16:20:48.789369 9307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:39547 every 8 connection(s)
I20260929 16:20:48.789963 9180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:48.790469 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9180
I20260929 16:20:48.790601 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:48.790884 4897 external_mini_cluster.cc:1428] Setting key 45b0bab3632381fc5ca59e9af9690dc1
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:48.803869 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:48.791732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:50505 (local address 127.4.200.126:35201)
0929 16:20:48.791869 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:48.791875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:48.792775 (+ 900us) server_negotiation.cc:407] Connection header received
0929 16:20:48.794091 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:48.794094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:48.794146 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:48.794225 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:48.795488 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.796030 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.797193 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.797442 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.797558 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:48.800173 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:48.800178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:48.800186 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:48.800211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:48.801616 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.802218 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.802223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.802225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.802348 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:48.802784 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:48.802787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:48.802789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:48.802976 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:48.803091 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:48.803581 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 16:20:48.803703 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:20:48.804708 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:48.792151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:35201 (local address 127.4.200.66:50505)
0929 16:20:48.792639 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:48.792680 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:48.793887 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:48.794439 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:48.794450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:48.794854 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:48.795341 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:48.795354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.796173 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.796176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:48.797062 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:48.797071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:48.797599 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:48.797604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:48.797777 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:48.798503 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:48.798519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:48.799993 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:48.801759 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:48.801767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:48.801784 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:48.802109 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:48.802481 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:48.802485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:48.802487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:48.802681 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:48.803081 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:48.803088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:48.803342 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 16:20:48.803538 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 16:20:48.803873 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":143,"threads_started":1}
I20260929 16:20:48.805894 9308 heartbeater.cc:347] Connected to a master server at 127.4.200.126:35201
I20260929 16:20:48.806156 9308 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:48.806686 9308 heartbeater.cc:511] Master 127.4.200.126:35201 requested a full tablet report, sending...
I20260929 16:20:48.807782 8985 ts_manager.cc:194] Registered new tserver with Master: 2bd1f99fff0043f69f148f85c09427f4 (127.4.200.66:39547)
I20260929 16:20:48.808619 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:50505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:48.817701 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:48.939123 9313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:48.939468 9313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:48.939563 9313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:48.946301 9313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:48.946425 9313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:48.946560 9313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:48.957470 9313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:35201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:48.959012 9313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:48.960275 9313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:48.969724 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.969693 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:48.969724 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:48.970229 9313 server_base.cc:1034] running on GCE node
I20260929 16:20:48.970737 9313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:48.971410 9313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:48.972620 9313 hybrid_clock.cc:648] HybridClock initialized: now 1790698848972601 us; error 34 us; skew 500 ppm
Sep 29 16:20:48 dist-test-slave-s9ct krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:48.975509 9313 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:48.976212 9313 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:48.976274 9313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:48.976451 9313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:48.978513 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "5a255fa89efa4b529333390d7b827c2d"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f615850b1aae413b7a70be83936d820"
server_key_iv: "15b8a62c88180e2490c2c1a2b09ffef2"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.979033 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "5a255fa89efa4b529333390d7b827c2d"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f615850b1aae413b7a70be83936d820"
server_key_iv: "15b8a62c88180e2490c2c1a2b09ffef2"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.982959 9313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 16:20:48.985292 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:48.986596 9313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:20:48.986689 9313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "5a255fa89efa4b529333390d7b827c2d"
format_stamp: "Formatted at 2026-09-29 16:20:48 on dist-test-slave-s9ct"
server_key: "6f615850b1aae413b7a70be83936d820"
server_key_iv: "15b8a62c88180e2490c2c1a2b09ffef2"
server_key_version: "encryptionkey@0"
I20260929 16:20:48.986773 9313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:49.011226 9313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:49.014806 9313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:49.014986 9313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:49.015559 9313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:49.016427 9313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:49.016480 9313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.016525 9313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:49.016543 9313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.025651 9313 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:45553
I20260929 16:20:49.025691 9440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:45553 every 8 connection(s)
I20260929 16:20:49.026261 9313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:49.034778 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9313
I20260929 16:20:49.034919 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:49.035264 4897 external_mini_cluster.cc:1428] Setting key 454b727a9b80ce399d8d21c2131cf20a
Sep 29 16:20:49 dist-test-slave-s9ct krb5kdc[8941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:49.041085 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.028106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:54973 (local address 127.4.200.126:35201)
0929 16:20:49.028271 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.028276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.029224 (+ 948us) server_negotiation.cc:407] Connection header received
0929 16:20:49.030751 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.030754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.030816 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.030893 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.032362 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.032924 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.034114 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.034358 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.034541 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.037547 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.037552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.037555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:49.037587 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.039018 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:49.039560 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:49.039563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:49.039565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:49.039655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:49.040032 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:49.040034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:49.040035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:49.040215 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.040293 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.040829 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.040959 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:20:49.041914 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.028564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:35201 (local address 127.4.200.67:54973)
0929 16:20:49.029059 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.029103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.030528 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.031065 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.031074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.031652 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:49.032182 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.032197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.033066 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.033071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.033971 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.033980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.034583 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.034588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.034864 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.035533 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:49.035551 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:49.037364 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:49.039136 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:49.039141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:49.039153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:49.039421 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:49.039788 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:49.039791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:49.039793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:49.039940 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:49.040321 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:49.040326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:49.040582 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.040749 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.041019 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260929 16:20:49.043211 9441 heartbeater.cc:347] Connected to a master server at 127.4.200.126:35201
I20260929 16:20:49.043505 9441 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:49.043991 9441 heartbeater.cc:511] Master 127.4.200.126:35201 requested a full tablet report, sending...
I20260929 16:20:49.045101 8985 ts_manager.cc:194] Registered new tserver with Master: 5a255fa89efa4b529333390d7b827c2d (127.4.200.67:45553)
I20260929 16:20:49.045691 8985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:54973
I20260929 16:20:49.049414 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:20:49.058718 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.051528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.4.200.126:35201)
0929 16:20:49.051663 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.051667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.051774 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:20:49.051871 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.051873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.051910 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.051977 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.052785 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.053283 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.054154 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.054365 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.054529 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.055954 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.055957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.055959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:49.055978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.057211 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:49.057584 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:49.057587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:49.057588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:49.057656 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:49.057869 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:49.057872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:49.057873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:49.058086 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.058188 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.058497 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.058594 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:20:49.061708 8985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:49.064915 8985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:49.080940 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.074195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.4.200.67:45553)
0929 16:20:49.076099 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.076111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.076134 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:20:49.076207 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.076213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.076397 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.076582 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:49.076827 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.078145 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.079506 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.080193 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080473 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080624 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.080680 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.080774 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1787,"thread_start_us":130,"threads_started":1}
I20260929 16:20:49.080932 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.073676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:45553 (local address 127.0.0.1:57200)
0929 16:20:49.074720 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.074734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.074812 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.076596 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.076598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.076616 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:49.076708 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.076714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.078296 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.078299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.079391 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.079398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.080322 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.080421 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080498 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.080584 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.080736 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":54,"thread_start_us":147,"threads_started":1}
I20260929 16:20:49.080976 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.075269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.4.200.65:46735)
0929 16:20:49.075968 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.075974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.075994 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:20:49.076053 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.076056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.076199 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.076375 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:49.076613 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.077407 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.079008 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.079876 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080001 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080307 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.080697 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.080795 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":102,"threads_started":1}
I20260929 16:20:49.081504 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.073412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:39547 (local address 127.0.0.1:51472)
0929 16:20:49.073917 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.073966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.074095 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.074987 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.074993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.075016 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:49.075922 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.075933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.077589 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.077593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.079093 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.079104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.080179 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.080364 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080460 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.080585 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.080738 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":127,"thread_start_us":75,"threads_started":1}
I20260929 16:20:49.080935 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.074181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:46735 (local address 127.0.0.1:33362)
0929 16:20:49.075003 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.075015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.075669 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.076351 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.076354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.076371 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:49.076475 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.076482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.077610 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.077615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.078875 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.078884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.080218 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.080378 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080481 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.080576 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.080804 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":252,"thread_start_us":90,"threads_started":1}
I20260929 16:20:49.082544 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.073677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.4.200.66:39547)
0929 16:20:49.074440 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.074453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.074478 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:20:49.074555 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.074561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.074812 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.076084 (+ 1272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:49.076246 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.077425 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.079220 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.080027 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.080127 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.080247 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.081477 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.081768 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":157,"threads_started":1}
I20260929 16:20:49.084275 9109 tablet_service.cc:1467] Processing CreateTablet for tablet b56428615e4742b08693de361c12bb7a (DEFAULT_TABLE table=test-table [id=a13e0e2cf46a4aa8821a0c5bbf1a67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:49.084317 9375 tablet_service.cc:1467] Processing CreateTablet for tablet b56428615e4742b08693de361c12bb7a (DEFAULT_TABLE table=test-table [id=a13e0e2cf46a4aa8821a0c5bbf1a67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:49.085289 9109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56428615e4742b08693de361c12bb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:49.085318 9375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56428615e4742b08693de361c12bb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:49.085005 9242 tablet_service.cc:1467] Processing CreateTablet for tablet b56428615e4742b08693de361c12bb7a (DEFAULT_TABLE table=test-table [id=a13e0e2cf46a4aa8821a0c5bbf1a67c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:49.085942 9242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56428615e4742b08693de361c12bb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:49.092927 9459 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Bootstrap starting.
I20260929 16:20:49.094079 9458 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Bootstrap starting.
I20260929 16:20:49.094866 9460 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: Bootstrap starting.
I20260929 16:20:49.096339 9458 tablet_bootstrap.cc:654] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:49.096544 9459 tablet_bootstrap.cc:654] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:49.097049 9460 tablet_bootstrap.cc:654] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:49.097903 9458 log.cc:824] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:49.097934 9459 log.cc:824] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:49.097934 9460 log.cc:824] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:49.100216 9459 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: No bootstrap required, opened a new log
I20260929 16:20:49.100217 9458 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: No bootstrap required, opened a new log
I20260929 16:20:49.100433 9458 ts_tablet_manager.cc:1397] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:20:49.100433 9459 ts_tablet_manager.cc:1397] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:20:49.101649 9460 tablet_bootstrap.cc:492] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: No bootstrap required, opened a new log
I20260929 16:20:49.101881 9460 ts_tablet_manager.cc:1397] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:20:49.104626 9459 raft_consensus.cc:348] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.104929 9459 raft_consensus.cc:374] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:49.104974 9459 raft_consensus.cc:729] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd1f99fff0043f69f148f85c09427f4, State: Initialized, Role: FOLLOWER
I20260929 16:20:49.105482 9459 consensus_queue.cc:260] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.104626 9458 raft_consensus.cc:348] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.105850 9458 raft_consensus.cc:374] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:49.105908 9458 raft_consensus.cc:729] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a255fa89efa4b529333390d7b827c2d, State: Initialized, Role: FOLLOWER
I20260929 16:20:49.106465 9458 consensus_queue.cc:260] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.106669 9459 ts_tablet_manager.cc:1428] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 16:20:49.107039 9308 heartbeater.cc:503] Master 127.4.200.126:35201 was elected leader, sending a full tablet report...
I20260929 16:20:49.107575 9460 raft_consensus.cc:348] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.107861 9460 raft_consensus.cc:374] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:49.107926 9460 raft_consensus.cc:729] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197e2b1d296842d58a2696922e25a398, State: Initialized, Role: FOLLOWER
I20260929 16:20:49.108379 9460 consensus_queue.cc:260] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.108704 9458 ts_tablet_manager.cc:1428] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:20:49.109084 9441 heartbeater.cc:503] Master 127.4.200.126:35201 was elected leader, sending a full tablet report...
I20260929 16:20:49.109084 9175 heartbeater.cc:503] Master 127.4.200.126:35201 was elected leader, sending a full tablet report...
I20260929 16:20:49.110631 9460 ts_tablet_manager.cc:1428] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 16:20:49.136626 9465 raft_consensus.cc:482] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:49.136888 9465 raft_consensus.cc:504] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.138123 9465 leader_election.cc:288] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bd1f99fff0043f69f148f85c09427f4 (127.4.200.66:39547), 197e2b1d296842d58a2696922e25a398 (127.4.200.65:46735)
I20260929 16:20:49.143540 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.138387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:46735 (local address 127.4.200.67:53443)
0929 16:20:49.138674 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.138690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.138794 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.139187 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.139193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.139217 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:49.139340 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.139347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.140599 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.140603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.142143 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.142150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.143154 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.143157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.143259 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.143291 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.143364 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.143414 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":101}
I20260929 16:20:49.143553 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.138422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:39547 (local address 127.4.200.67:43419)
0929 16:20:49.139088 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.139102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.139232 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.139497 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.139500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.139527 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:49.139661 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.139667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.140599 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.140603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.142139 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:49.142150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.143044 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.143047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:49.143173 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.143200 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.143295 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.143383 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":148,"thread_start_us":121,"threads_started":1}
I20260929 16:20:49.143615 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.138557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:53443 (local address 127.4.200.65:46735)
0929 16:20:49.138674 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.138679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.138740 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:20:49.138916 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.138922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.138974 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.139053 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:49.139605 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.140441 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.142277 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.143032 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.143155 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.143202 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.143415 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.143493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:20:49.143607 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.138496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:43419 (local address 127.4.200.66:39547)
0929 16:20:49.138792 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.138796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.139168 (+ 372us) server_negotiation.cc:407] Connection header received
0929 16:20:49.139348 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.139352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.139410 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.139486 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:49.139777 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.140473 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.142276 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.142912 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.143036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.143106 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.143364 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.143469 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260929 16:20:49.144273 9262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56428615e4742b08693de361c12bb7a" candidate_uuid: "5a255fa89efa4b529333390d7b827c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd1f99fff0043f69f148f85c09427f4" is_pre_election: true
I20260929 16:20:49.144290 9129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56428615e4742b08693de361c12bb7a" candidate_uuid: "5a255fa89efa4b529333390d7b827c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197e2b1d296842d58a2696922e25a398" is_pre_election: true
I20260929 16:20:49.144584 9129 raft_consensus.cc:2454] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a255fa89efa4b529333390d7b827c2d in term 0.
I20260929 16:20:49.144587 9262 raft_consensus.cc:2454] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a255fa89efa4b529333390d7b827c2d in term 0.
I20260929 16:20:49.145121 9329 leader_election.cc:302] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 197e2b1d296842d58a2696922e25a398, 2bd1f99fff0043f69f148f85c09427f4, 5a255fa89efa4b529333390d7b827c2d; no voters:
I20260929 16:20:49.145428 9465 raft_consensus.cc:2781] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:49.145555 9465 raft_consensus.cc:482] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:49.145617 9465 raft_consensus.cc:3037] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:49.149964 9465 raft_consensus.cc:504] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.150476 9465 leader_election.cc:288] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [CANDIDATE]: Term 1 election: Requested vote from peers 2bd1f99fff0043f69f148f85c09427f4 (127.4.200.66:39547), 197e2b1d296842d58a2696922e25a398 (127.4.200.65:46735)
I20260929 16:20:49.150900 9262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56428615e4742b08693de361c12bb7a" candidate_uuid: "5a255fa89efa4b529333390d7b827c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd1f99fff0043f69f148f85c09427f4"
I20260929 16:20:49.150911 9129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56428615e4742b08693de361c12bb7a" candidate_uuid: "5a255fa89efa4b529333390d7b827c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197e2b1d296842d58a2696922e25a398"
I20260929 16:20:49.151060 9262 raft_consensus.cc:3037] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:49.151085 9129 raft_consensus.cc:3037] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:49.152199 9262 raft_consensus.cc:2454] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a255fa89efa4b529333390d7b827c2d in term 1.
I20260929 16:20:49.152565 9329 leader_election.cc:302] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd1f99fff0043f69f148f85c09427f4, 5a255fa89efa4b529333390d7b827c2d; no voters:
I20260929 16:20:49.152670 9129 raft_consensus.cc:2454] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a255fa89efa4b529333390d7b827c2d in term 1.
I20260929 16:20:49.152753 9465 raft_consensus.cc:2781] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:49.153021 9465 raft_consensus.cc:686] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 1 LEADER]: Becoming Leader. State: Replica: 5a255fa89efa4b529333390d7b827c2d, State: Running, Role: LEADER
I20260929 16:20:49.153417 9465 consensus_queue.cc:237] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } }
I20260929 16:20:49.157383 8983 catalog_manager.cc:5495] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a255fa89efa4b529333390d7b827c2d (127.4.200.67). New cstate: current_term: 1 leader_uuid: "5a255fa89efa4b529333390d7b827c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a255fa89efa4b529333390d7b827c2d" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 } health_report { overall_health: UNKNOWN } } }
I20260929 16:20:49.184093 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.180506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.4.200.67:45553)
0929 16:20:49.180634 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.180637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.180651 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:49.180775 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.180778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.180832 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.180922 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:49.181148 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.181761 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.182650 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:49.182859 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:49.182947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:49.183248 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:49.183736 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:49.183828 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.183909 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.183961 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:20:49.190047 9262 raft_consensus.cc:1261] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 1 FOLLOWER]: Refusing update from remote peer 5a255fa89efa4b529333390d7b827c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:49.190047 9129 raft_consensus.cc:1261] T b56428615e4742b08693de361c12bb7a P 197e2b1d296842d58a2696922e25a398 [term 1 FOLLOWER]: Refusing update from remote peer 5a255fa89efa4b529333390d7b827c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:49.190811 9465 consensus_queue.cc:1035] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "197e2b1d296842d58a2696922e25a398" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:49.191048 9468 consensus_queue.cc:1035] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bd1f99fff0043f69f148f85c09427f4" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:49.199239 9473 mvcc.cc:205] Tried to move back new op lower bound from 7334702486275051520 to 7334702486135779328. Current Snapshot: MvccSnapshot[applied={T|T < 7334702486275051520}]
I20260929 16:20:49.201062 9472 mvcc.cc:205] Tried to move back new op lower bound from 7334702486275051520 to 7334702486135779328. Current Snapshot: MvccSnapshot[applied={T|T < 7334702486275051520 or (T in {7334702486275051520})}]
I20260929 16:20:49.207162 8983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:49.207355 8983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:49.209798 8983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9774e70efa3e4f77811f74053e28def1: Sending DeleteTablet for 3 replicas of tablet b56428615e4742b08693de361c12bb7a
I20260929 16:20:49.210677 9242 tablet_service.cc:1514] Processing DeleteTablet for tablet b56428615e4742b08693de361c12bb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:51472
I20260929 16:20:49.210748 9109 tablet_service.cc:1514] Processing DeleteTablet for tablet b56428615e4742b08693de361c12bb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:33362
I20260929 16:20:49.211354 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9047
I20260929 16:20:49.211421 9375 tablet_service.cc:1514] Processing DeleteTablet for tablet b56428615e4742b08693de361c12bb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:57200
I20260929 16:20:49.212138 9481 tablet_replica.cc:330] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: stopping tablet replica
I20260929 16:20:49.212520 9481 raft_consensus.cc:2229] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:49.212934 9481 raft_consensus.cc:2258] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:49.214080 9483 tablet_replica.cc:330] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: stopping tablet replica
I20260929 16:20:49.214417 9483 raft_consensus.cc:2229] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:49.214797 9483 raft_consensus.cc:2258] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:49.215207 9481 ts_tablet_manager.cc:1905] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:49.216264 9483 ts_tablet_manager.cc:1905] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:49.218027 9481 ts_tablet_manager.cc:1918] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:49.218176 9481 log.cc:1196] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/b56428615e4742b08693de361c12bb7a
I20260929 16:20:49.218590 9481 ts_tablet_manager.cc:1939] T b56428615e4742b08693de361c12bb7a P 2bd1f99fff0043f69f148f85c09427f4: Deleting consensus metadata
I20260929 16:20:49.219565 9483 ts_tablet_manager.cc:1918] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:49.219664 9483 log.cc:1196] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/b56428615e4742b08693de361c12bb7a
W20260929 16:20:49.219841 8972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:46735 (error 108)
I20260929 16:20:49.220012 9483 ts_tablet_manager.cc:1939] T b56428615e4742b08693de361c12bb7a P 5a255fa89efa4b529333390d7b827c2d: Deleting consensus metadata
I20260929 16:20:49.220008 8970 catalog_manager.cc:4841] TS 2bd1f99fff0043f69f148f85c09427f4 (127.4.200.66:39547): tablet b56428615e4742b08693de361c12bb7a (table test-table [id=a13e0e2cf46a4aa8821a0c5bbf1a67c5]) successfully deleted
I20260929 16:20:49.220691 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9180
I20260929 16:20:49.221053 8969 catalog_manager.cc:4841] TS 5a255fa89efa4b529333390d7b827c2d (127.4.200.67:45553): tablet b56428615e4742b08693de361c12bb7a (table test-table [id=a13e0e2cf46a4aa8821a0c5bbf1a67c5]) successfully deleted
I20260929 16:20:49.223527 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.223168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:46735 (local address 127.0.0.1:33374)
0929 16:20:49.223338 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.223423 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:20:49.223738 8972 catalog_manager.cc:4584] TS 197e2b1d296842d58a2696922e25a398 (127.4.200.65:46735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b56428615e4742b08693de361c12bb7a: Network error: Client connection negotiation failed: client connection to 127.4.200.65:46735: connect: Connection refused (error 111)
I20260929 16:20:49.226974 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9313
I20260929 16:20:49.231805 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 8954
2026-09-29T16:20:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2341 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:49Z Disabled control of system clock
I20260929 16:20:49.261896 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43379
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:43379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:20:49.372144 9488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:49.372416 9488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:49.372453 9488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:49.376793 9488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:49.376858 9488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:49.376874 9488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:49.376892 9488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:49.376907 9488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:49.381763 9488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:43379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:49.382910 9488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:49.384330 9488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:49.392829 9496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.392839 9493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.392845 9494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.393245 9488 server_base.cc:1034] running on GCE node
I20260929 16:20:49.393783 9488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:49.394949 9488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:49.396162 9488 hybrid_clock.cc:648] HybridClock initialized: now 1790698849396138 us; error 33 us; skew 500 ppm
I20260929 16:20:49.398150 9488 webserver.cc:460] Webserver started at http://127.4.200.126:42163/ using document root <none> and password file <none>
I20260929 16:20:49.398816 9488 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:49.398918 9488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:49.399168 9488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:49.401299 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "71f71ca4efee4318b353896a64444de1"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "0827d3a043c08a8e16b52959c99a7e79"
server_key_iv: "325708743fb94eced02643fefadb0820"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.401928 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "71f71ca4efee4318b353896a64444de1"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "0827d3a043c08a8e16b52959c99a7e79"
server_key_iv: "325708743fb94eced02643fefadb0820"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.405848 9488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:20:49.408474 9502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.409802 9488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:49.409946 9488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "71f71ca4efee4318b353896a64444de1"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "0827d3a043c08a8e16b52959c99a7e79"
server_key_iv: "325708743fb94eced02643fefadb0820"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.410071 9488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:49.414779 9488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:49.415432 9488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:49.415627 9488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:49.422169 9488 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:43379
I20260929 16:20:49.422180 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:43379 every 8 connection(s)
I20260929 16:20:49.423164 9488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:49.426224 9555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:49.427843 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9488
I20260929 16:20:49.427980 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:49.428259 4897 external_mini_cluster.cc:1428] Setting key 220df98a69eaa0a43c9f0373e3b05453
I20260929 16:20:49.432667 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: Bootstrap starting.
I20260929 16:20:49.433771 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.429675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.4.200.126:43379)
0929 16:20:49.430314 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.430328 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.430362 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:20:49.431058 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.431082 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.431422 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.431757 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.432547 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.432551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.432562 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:49.432574 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.432628 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:49.432643 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.432791 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.432862 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.433105 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260929 16:20:49.435191 9555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:49.436139 9555 log.cc:824] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:49.437949 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: No bootstrap required, opened a new log
I20260929 16:20:49.441326 9555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } }
I20260929 16:20:49.441581 9555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:49.441643 9555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f71ca4efee4318b353896a64444de1, State: Initialized, Role: FOLLOWER
I20260929 16:20:49.442085 9555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } }
I20260929 16:20:49.442211 9555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:49.442260 9555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:49.442327 9555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:49.443193 9555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } }
I20260929 16:20:49.443531 9555 leader_election.cc:302] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71f71ca4efee4318b353896a64444de1; no voters:
I20260929 16:20:49.443825 9555 leader_election.cc:288] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:49.443964 9560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:49.444212 9560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [term 1 LEADER]: Becoming Leader. State: Replica: 71f71ca4efee4318b353896a64444de1, State: Running, Role: LEADER
I20260929 16:20:49.444540 9560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } }
I20260929 16:20:49.445158 9555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:49.446349 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71f71ca4efee4318b353896a64444de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } } }
I20260929 16:20:49.446349 9562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f71ca4efee4318b353896a64444de1. Latest consensus state: current_term: 1 leader_uuid: "71f71ca4efee4318b353896a64444de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f71ca4efee4318b353896a64444de1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43379 } } }
I20260929 16:20:49.446518 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:49.446518 9562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:49.447546 9569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:49.450085 9569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:49.455029 9569 catalog_manager.cc:1347] Generated new cluster ID: 6b5d95bad7ab4d3eb842cf6c92779471
I20260929 16:20:49.455090 9569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:49.463723 9569 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:49.464023 9569 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:49.474924 9569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: Generated new TSK 0
I20260929 16:20:49.475612 9569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:20:49.489018 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--builtin_ntp_servers=127.4.200.84:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:20:49.599690 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:49.599969 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:49.600041 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:49.604202 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:49.604259 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:49.604314 9579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:49.609021 9579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:49.610098 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:49.611529 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:49.621104 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.621169 9579 server_base.cc:1034] running on GCE node
W20260929 16:20:49.621104 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.621132 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.621796 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:49.622448 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:49.623634 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790698849623617 us; error 36 us; skew 500 ppm
I20260929 16:20:49.625525 9579 webserver.cc:460] Webserver started at http://127.4.200.65:43969/ using document root <none> and password file <none>
I20260929 16:20:49.626124 9579 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:49.626187 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:49.626498 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:49.628520 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "5df04801d1a34a80a3f4062aca93cb06"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "6ee10d206e7f09f78d21ca5200f0cac9"
server_key_iv: "79f3cf8cda218072382146a66e5c810c"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.629081 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "5df04801d1a34a80a3f4062aca93cb06"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "6ee10d206e7f09f78d21ca5200f0cac9"
server_key_iv: "79f3cf8cda218072382146a66e5c810c"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.632658 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:20:49.635030 9593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.636332 9579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:49.636456 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "5df04801d1a34a80a3f4062aca93cb06"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "6ee10d206e7f09f78d21ca5200f0cac9"
server_key_iv: "79f3cf8cda218072382146a66e5c810c"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.636561 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:49.641023 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:49.641592 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:49.641767 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:49.642314 9579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:49.643209 9579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:49.643255 9579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.643321 9579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:49.643353 9579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.652225 9579 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:45701
I20260929 16:20:49.652242 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:45701 every 8 connection(s)
I20260929 16:20:49.653133 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:49.655195 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9579
I20260929 16:20:49.655329 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:49.655618 4897 external_mini_cluster.cc:1428] Setting key 44cb270a445523dda70be0782adae0e3
I20260929 16:20:49.657433 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--builtin_ntp_servers=127.4.200.84:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:20:49.659523 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.655021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:51467 (local address 127.4.200.126:43379)
0929 16:20:49.655184 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.655189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.656330 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 16:20:49.657765 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.657771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.657817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.657908 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.659122 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.659125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.659127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:49.659137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.659154 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:49.659162 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.659231 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.659391 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.659425 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:20:49.660524 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.655510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43379 (local address 127.4.200.65:51467)
0929 16:20:49.656200 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.656249 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.657513 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.658057 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.658064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.658160 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:49.658805 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:49.658810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:49.658874 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:49.658966 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:49.659247 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:49.659257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:49.659264 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.659439 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.659693 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260929 16:20:49.661695 9707 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43379
I20260929 16:20:49.661933 9707 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:49.662317 9707 heartbeater.cc:511] Master 127.4.200.126:43379 requested a full tablet report, sending...
I20260929 16:20:49.663982 9519 ts_manager.cc:194] Registered new tserver with Master: 5df04801d1a34a80a3f4062aca93cb06 (127.4.200.65:45701)
W20260929 16:20:49.768075 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:49.768316 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:49.768352 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:49.772207 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:49.772251 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:49.772305 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:49.776878 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:49.777802 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:49.779090 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:49.788466 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.788538 9710 server_base.cc:1034] running on GCE node
W20260929 16:20:49.788587 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.788724 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.789220 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:49.789928 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:49.791107 9710 hybrid_clock.cc:648] HybridClock initialized: now 1790698849791079 us; error 31 us; skew 500 ppm
I20260929 16:20:49.793063 9710 webserver.cc:460] Webserver started at http://127.4.200.66:41665/ using document root <none> and password file <none>
I20260929 16:20:49.793671 9710 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:49.793741 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:49.793968 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:49.796149 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "a12b10eefe6240d687eabbf214e35ee0"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "853c6b2b8541e0cfc6a6480bb8a479d6"
server_key_iv: "f4fddfc816e209f46adb8bd6c9438d5e"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.796744 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "a12b10eefe6240d687eabbf214e35ee0"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "853c6b2b8541e0cfc6a6480bb8a479d6"
server_key_iv: "f4fddfc816e209f46adb8bd6c9438d5e"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.800626 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:20:49.803174 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.804451 9710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:20:49.804596 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "a12b10eefe6240d687eabbf214e35ee0"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "853c6b2b8541e0cfc6a6480bb8a479d6"
server_key_iv: "f4fddfc816e209f46adb8bd6c9438d5e"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.804721 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:49.809532 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:49.810168 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:49.810369 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:49.811000 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:49.811933 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:49.812008 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.812083 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:49.812140 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.821681 9710 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:46415
I20260929 16:20:49.821699 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:46415 every 8 connection(s)
I20260929 16:20:49.822729 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:49.823972 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9710
I20260929 16:20:49.824095 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:49.824383 4897 external_mini_cluster.cc:1428] Setting key af164101af6bcae5ec8c6221928e53fc
I20260929 16:20:49.826733 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--builtin_ntp_servers=127.4.200.84:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:20:49.830307 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.824784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:33863 (local address 127.4.200.126:43379)
0929 16:20:49.824939 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.824943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.826357 (+ 1414us) server_negotiation.cc:407] Connection header received
0929 16:20:49.828269 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.828273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.828329 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.828462 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.829723 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.829726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.829729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:49.829743 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.829783 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:49.829801 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.829937 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:49.830100 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:20:49.830153 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:20:49.831223 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.825302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43379 (local address 127.4.200.66:33863)
0929 16:20:49.826188 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.826242 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.827995 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.828663 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.828676 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.828779 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:49.829450 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:49.829451 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:49.829493 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:49.829562 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:49.829926 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:49.829935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:49.829943 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:20:49.830149 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 16:20:49.830456 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
I20260929 16:20:49.832603 9838 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43379
I20260929 16:20:49.832867 9838 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:49.833293 9838 heartbeater.cc:511] Master 127.4.200.126:43379 requested a full tablet report, sending...
I20260929 16:20:49.834688 9519 ts_manager.cc:194] Registered new tserver with Master: a12b10eefe6240d687eabbf214e35ee0 (127.4.200.66:46415)
W20260929 16:20:49.940171 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:49.940462 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:49.940531 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:49.944692 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:49.944777 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:49.944865 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:49.950166 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:49.951314 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:49.952729 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:49.962321 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.962301 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:49.962301 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:49.962345 9841 server_base.cc:1034] running on GCE node
I20260929 16:20:49.962903 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:49.963552 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:49.964757 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790698849964722 us; error 51 us; skew 500 ppm
I20260929 16:20:49.966624 9841 webserver.cc:460] Webserver started at http://127.4.200.67:33433/ using document root <none> and password file <none>
I20260929 16:20:49.967193 9841 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:49.967283 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:49.967514 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:49.969496 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "b011106702c34b6b97169eb89dd73532"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "ca2acbb8483781d4e6ff8c3f297466c3"
server_key_iv: "223e34d0a72188a79faf92c5970ca153"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.970054 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "b011106702c34b6b97169eb89dd73532"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "ca2acbb8483781d4e6ff8c3f297466c3"
server_key_iv: "223e34d0a72188a79faf92c5970ca153"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.973793 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:20:49.976248 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.977516 9841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:20:49.977675 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "b011106702c34b6b97169eb89dd73532"
format_stamp: "Formatted at 2026-09-29 16:20:49 on dist-test-slave-s9ct"
server_key: "ca2acbb8483781d4e6ff8c3f297466c3"
server_key_iv: "223e34d0a72188a79faf92c5970ca153"
server_key_version: "encryptionkey@0"
I20260929 16:20:49.977794 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:49.982523 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:49.983160 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:49.983350 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:49.983943 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:49.984926 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:49.984992 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.985062 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:49.985113 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:49.993491 9841 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:43999
I20260929 16:20:49.993553 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:43999 every 8 connection(s)
I20260929 16:20:49.994444 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:50.000244 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.996171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:60489 (local address 127.4.200.126:43379)
0929 16:20:49.996340 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:49.996344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:49.997366 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 16:20:49.998649 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:49.998651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:49.998710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:49.998795 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:49.999890 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:49.999892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:49.999894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:49.999903 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:49.999916 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:49.999923 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:49.999990 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.000098 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.000136 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:20:50.001037 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:49.996593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43379 (local address 127.4.200.67:60489)
0929 16:20:49.997214 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:49.997252 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:49.998457 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:49.998951 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:49.998957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:49.999056 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:49.999658 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:49.999659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:49.999698 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:49.999764 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:49.999980 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:49.999984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:49.999987 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.000104 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.000282 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260929 16:20:50.002135 9969 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43379
I20260929 16:20:50.002360 9969 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:50.002787 9969 heartbeater.cc:511] Master 127.4.200.126:43379 requested a full tablet report, sending...
I20260929 16:20:50.003787 9519 ts_manager.cc:194] Registered new tserver with Master: b011106702c34b6b97169eb89dd73532 (127.4.200.67:43999)
I20260929 16:20:50.004019 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 9841
I20260929 16:20:50.004161 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:50.004457 4897 external_mini_cluster.cc:1428] Setting key e000e192621dabfeccd5a615035e4ce9
I20260929 16:20:50.006307 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:20:50.009908 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.008231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.4.200.126:43379)
0929 16:20:50.008377 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.008382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.008531 (+ 149us) server_negotiation.cc:407] Connection header received
0929 16:20:50.008679 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.008683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.008725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.008822 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.009569 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.009581 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.009583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.009592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.009608 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.009615 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.009679 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.009723 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.009753 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:20:50.012475 9519 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:50.015506 9519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:50.026948 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.023714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46415 (local address 127.0.0.1:57676)
0929 16:20:50.024373 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.024387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.024464 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.025765 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.025768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.025783 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:50.026198 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:50.026200 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:50.026210 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:50.026236 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:50.026593 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:50.026605 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:50.026614 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.026725 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.026795 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":102,"threads_started":1}
I20260929 16:20:50.026995 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.024190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.4.200.66:46415)
0929 16:20:50.024645 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.024658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.024682 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:20:50.024737 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.024743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.024944 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.025097 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.026361 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.026367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.026377 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.026388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.026483 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.026506 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.026593 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.026767 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.026836 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":134,"threads_started":1}
I20260929 16:20:50.026952 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.023558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:45701 (local address 127.0.0.1:55656)
0929 16:20:50.024039 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.024088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.024200 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.025113 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.025119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.025193 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:50.025734 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:50.025736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:50.025757 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:50.025820 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:50.026674 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:50.026677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:50.026680 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.026765 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.026803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":65,"threads_started":1}
I20260929 16:20:50.027460 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.024239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43999 (local address 127.0.0.1:50464)
0929 16:20:50.024551 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.024566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.024640 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.025729 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.025735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.025754 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:50.026671 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:50.026673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:50.026682 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:50.026707 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:50.027212 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:50.027215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:50.027218 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.027283 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.027341 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260929 16:20:50.027912 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.023707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.4.200.65:45701)
0929 16:20:50.024716 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.024727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.024745 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:20:50.024789 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.024793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.024943 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.025478 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.026463 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.026468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.026479 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.026490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.026527 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.026563 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.026683 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.027479 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.027751 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":116,"threads_started":1}
I20260929 16:20:50.027942 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.024493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.4.200.67:43999)
0929 16:20:50.025030 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.025042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.025065 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:20:50.025115 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.025121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.025302 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.025437 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.026837 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.026843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.026853 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.026865 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.026907 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.026934 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.027671 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.027726 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.027784 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":86,"threads_started":1}
I20260929 16:20:50.029989 9772 tablet_service.cc:1467] Processing CreateTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 (DEFAULT_TABLE table=test-table [id=cdd0fdf6f26f41bfa7b1e6212890f82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:50.030557 9903 tablet_service.cc:1467] Processing CreateTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 (DEFAULT_TABLE table=test-table [id=cdd0fdf6f26f41bfa7b1e6212890f82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:50.030525 9641 tablet_service.cc:1467] Processing CreateTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 (DEFAULT_TABLE table=test-table [id=cdd0fdf6f26f41bfa7b1e6212890f82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:50.031128 9772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4cb8fb2e5d457cb5dabaf1649babc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:50.031446 9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4cb8fb2e5d457cb5dabaf1649babc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:50.031450 9641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4cb8fb2e5d457cb5dabaf1649babc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:50.037309 9986 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Bootstrap starting.
I20260929 16:20:50.039146 9987 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Bootstrap starting.
I20260929 16:20:50.039888 9986 tablet_bootstrap.cc:654] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:50.040609 9986 log.cc:824] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:50.041100 9987 tablet_bootstrap.cc:654] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:50.041275 9988 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: Bootstrap starting.
I20260929 16:20:50.041942 9987 log.cc:824] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:50.042783 9986 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: No bootstrap required, opened a new log
I20260929 16:20:50.042979 9986 ts_tablet_manager.cc:1397] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:20:50.043306 9988 tablet_bootstrap.cc:654] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:50.043936 9987 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: No bootstrap required, opened a new log
I20260929 16:20:50.044106 9987 ts_tablet_manager.cc:1397] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:20:50.044143 9988 log.cc:824] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:50.045948 9988 tablet_bootstrap.cc:492] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: No bootstrap required, opened a new log
I20260929 16:20:50.046154 9988 ts_tablet_manager.cc:1397] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:20:50.047705 9986 raft_consensus.cc:348] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.047997 9986 raft_consensus.cc:374] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:50.048054 9986 raft_consensus.cc:729] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b011106702c34b6b97169eb89dd73532, State: Initialized, Role: FOLLOWER
I20260929 16:20:50.048537 9986 consensus_queue.cc:260] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.048589 9987 raft_consensus.cc:348] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.048830 9987 raft_consensus.cc:374] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:50.048890 9987 raft_consensus.cc:729] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12b10eefe6240d687eabbf214e35ee0, State: Initialized, Role: FOLLOWER
I20260929 16:20:50.049299 9987 consensus_queue.cc:260] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.049649 9986 ts_tablet_manager.cc:1428] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:20:50.049983 9969 heartbeater.cc:503] Master 127.4.200.126:43379 was elected leader, sending a full tablet report...
I20260929 16:20:50.050184 9987 ts_tablet_manager.cc:1428] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:20:50.050309 9988 raft_consensus.cc:348] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.050552 9838 heartbeater.cc:503] Master 127.4.200.126:43379 was elected leader, sending a full tablet report...
I20260929 16:20:50.050590 9988 raft_consensus.cc:374] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:50.050637 9988 raft_consensus.cc:729] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df04801d1a34a80a3f4062aca93cb06, State: Initialized, Role: FOLLOWER
I20260929 16:20:50.051066 9988 consensus_queue.cc:260] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.051680 9707 heartbeater.cc:503] Master 127.4.200.126:43379 was elected leader, sending a full tablet report...
I20260929 16:20:50.052445 9988 ts_tablet_manager.cc:1428] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 16:20:50.073765 9839 tablet.cc:2369] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:50.104509 9993 raft_consensus.cc:482] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:50.104761 9993 raft_consensus.cc:504] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.105904 9993 leader_election.cc:288] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5df04801d1a34a80a3f4062aca93cb06 (127.4.200.65:45701), b011106702c34b6b97169eb89dd73532 (127.4.200.67:43999)
I20260929 16:20:50.107877 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.106160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:45701 (local address 127.4.200.66:38857)
0929 16:20:50.106297 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.106314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.106412 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.106672 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.106675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.106692 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:50.107234 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:50.107236 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:50.107247 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:50.107276 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:50.107614 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:50.107619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:50.107622 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.107700 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.107742 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:20:50.107944 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.106282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:38857 (local address 127.4.200.65:45701)
0929 16:20:50.106439 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.106445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.106462 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:20:50.106511 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.106514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.106551 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.106638 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.107432 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.107437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.107443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.107457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.107481 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.107493 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.107596 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.107718 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.107791 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:50.108589 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.106434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:43999 (local address 127.4.200.66:42579)
0929 16:20:50.106848 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.106859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.106922 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.107226 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.107229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.107244 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:50.107788 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:20:50.107790 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:50.107800 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:20:50.107822 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:50.108331 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:50.108333 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:50.108336 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:20:50.108410 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:20:50.108442 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":176,"threads_started":1}
I20260929 16:20:50.108625 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.106547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:42579 (local address 127.4.200.67:43999)
0929 16:20:50.106986 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.106991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.107007 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:20:50.107054 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.107059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.107098 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.107182 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.107926 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.107929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.107932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.107944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.107967 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.107979 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.108049 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.108422 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.108482 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260929 16:20:50.108551 9661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4cb8fb2e5d457cb5dabaf1649babc7" candidate_uuid: "a12b10eefe6240d687eabbf214e35ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df04801d1a34a80a3f4062aca93cb06" is_pre_election: true
I20260929 16:20:50.108934 9661 raft_consensus.cc:2454] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12b10eefe6240d687eabbf214e35ee0 in term 0.
I20260929 16:20:50.109258 9923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4cb8fb2e5d457cb5dabaf1649babc7" candidate_uuid: "a12b10eefe6240d687eabbf214e35ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b011106702c34b6b97169eb89dd73532" is_pre_election: true
I20260929 16:20:50.109450 9726 leader_election.cc:302] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df04801d1a34a80a3f4062aca93cb06, a12b10eefe6240d687eabbf214e35ee0; no voters:
I20260929 16:20:50.109568 9923 raft_consensus.cc:2454] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12b10eefe6240d687eabbf214e35ee0 in term 0.
I20260929 16:20:50.109740 9993 raft_consensus.cc:2781] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:50.109838 9993 raft_consensus.cc:482] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:50.109899 9993 raft_consensus.cc:3037] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:50.111019 9993 raft_consensus.cc:504] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.111349 9993 leader_election.cc:288] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [CANDIDATE]: Term 1 election: Requested vote from peers 5df04801d1a34a80a3f4062aca93cb06 (127.4.200.65:45701), b011106702c34b6b97169eb89dd73532 (127.4.200.67:43999)
I20260929 16:20:50.111724 9661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4cb8fb2e5d457cb5dabaf1649babc7" candidate_uuid: "a12b10eefe6240d687eabbf214e35ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df04801d1a34a80a3f4062aca93cb06"
I20260929 16:20:50.111763 9923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4cb8fb2e5d457cb5dabaf1649babc7" candidate_uuid: "a12b10eefe6240d687eabbf214e35ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b011106702c34b6b97169eb89dd73532"
I20260929 16:20:50.111873 9661 raft_consensus.cc:3037] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:50.111879 9923 raft_consensus.cc:3037] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:50.112932 9661 raft_consensus.cc:2454] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12b10eefe6240d687eabbf214e35ee0 in term 1.
I20260929 16:20:50.112993 9923 raft_consensus.cc:2454] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12b10eefe6240d687eabbf214e35ee0 in term 1.
I20260929 16:20:50.113278 9726 leader_election.cc:302] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df04801d1a34a80a3f4062aca93cb06, a12b10eefe6240d687eabbf214e35ee0; no voters:
I20260929 16:20:50.113494 9993 raft_consensus.cc:2781] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:50.113768 9993 raft_consensus.cc:686] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 1 LEADER]: Becoming Leader. State: Replica: a12b10eefe6240d687eabbf214e35ee0, State: Running, Role: LEADER
I20260929 16:20:50.114166 9993 consensus_queue.cc:237] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } }
I20260929 16:20:50.117537 9519 catalog_manager.cc:5495] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a12b10eefe6240d687eabbf214e35ee0 (127.4.200.66). New cstate: current_term: 1 leader_uuid: "a12b10eefe6240d687eabbf214e35ee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a12b10eefe6240d687eabbf214e35ee0" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260929 16:20:50.130093 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.128642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.4.200.66:46415)
0929 16:20:50.128765 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.128769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.128782 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:20:50.128901 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.128903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.128958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.129028 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.129709 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.129712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.129714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.129721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.129735 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.129741 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.129806 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.129884 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.129917 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:20:50.135653 9661 raft_consensus.cc:1261] T fc4cb8fb2e5d457cb5dabaf1649babc7 P 5df04801d1a34a80a3f4062aca93cb06 [term 1 FOLLOWER]: Refusing update from remote peer a12b10eefe6240d687eabbf214e35ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:50.135691 9923 raft_consensus.cc:1261] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 1 FOLLOWER]: Refusing update from remote peer a12b10eefe6240d687eabbf214e35ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:50.136381 9996 consensus_queue.cc:1035] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df04801d1a34a80a3f4062aca93cb06" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:50.136545 9993 consensus_queue.cc:1035] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b011106702c34b6b97169eb89dd73532" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:50.152726 10000 mvcc.cc:205] Tried to move back new op lower bound from 7334702490148560896 to 7334702490070671360. Current Snapshot: MvccSnapshot[applied={T|T < 7334702490148560896}]
I20260929 16:20:50.159644 9519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:50.159821 9519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:50.162058 9519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71f71ca4efee4318b353896a64444de1: Sending DeleteTablet for 3 replicas of tablet fc4cb8fb2e5d457cb5dabaf1649babc7
I20260929 16:20:50.162832 9641 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:50 UTC) from {username='slave'} at 127.0.0.1:55656
I20260929 16:20:50.162886 9903 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:50 UTC) from {username='slave'} at 127.0.0.1:50464
I20260929 16:20:50.162959 9772 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4cb8fb2e5d457cb5dabaf1649babc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:50 UTC) from {username='slave'} at 127.0.0.1:57676
I20260929 16:20:50.163360 10009 tablet_replica.cc:330] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: stopping tablet replica
I20260929 16:20:50.163360 10010 tablet_replica.cc:330] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: stopping tablet replica
I20260929 16:20:50.163662 10009 raft_consensus.cc:2229] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:50.163662 10010 raft_consensus.cc:2229] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:50.163976 10009 raft_consensus.cc:2258] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:50.164011 10010 raft_consensus.cc:2258] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:50.164052 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9579
I20260929 16:20:50.165477 10010 ts_tablet_manager.cc:1905] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:50.166198 10009 ts_tablet_manager.cc:1905] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:50.168555 10010 ts_tablet_manager.cc:1918] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:50.168658 10010 log.cc:1196] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/fc4cb8fb2e5d457cb5dabaf1649babc7
I20260929 16:20:50.169006 10010 ts_tablet_manager.cc:1939] T fc4cb8fb2e5d457cb5dabaf1649babc7 P a12b10eefe6240d687eabbf214e35ee0: Deleting consensus metadata
I20260929 16:20:50.169112 10009 ts_tablet_manager.cc:1918] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:50.169214 10009 log.cc:1196] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/fc4cb8fb2e5d457cb5dabaf1649babc7
I20260929 16:20:50.169510 10009 ts_tablet_manager.cc:1939] T fc4cb8fb2e5d457cb5dabaf1649babc7 P b011106702c34b6b97169eb89dd73532: Deleting consensus metadata
I20260929 16:20:50.170258 9506 catalog_manager.cc:4841] TS a12b10eefe6240d687eabbf214e35ee0 (127.4.200.66:46415): tablet fc4cb8fb2e5d457cb5dabaf1649babc7 (table test-table [id=cdd0fdf6f26f41bfa7b1e6212890f82a]) successfully deleted
I20260929 16:20:50.170634 9504 catalog_manager.cc:4841] TS b011106702c34b6b97169eb89dd73532 (127.4.200.67:43999): tablet fc4cb8fb2e5d457cb5dabaf1649babc7 (table test-table [id=cdd0fdf6f26f41bfa7b1e6212890f82a]) successfully deleted
W20260929 16:20:50.174263 9504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:45701 (error 108)
I20260929 16:20:50.174322 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9710
I20260929 16:20:50.175750 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.175439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:45701 (local address 127.0.0.1:55666)
0929 16:20:50.175604 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.175667 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:45701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 16:20:50.176004 9504 catalog_manager.cc:4584] TS 5df04801d1a34a80a3f4062aca93cb06 (127.4.200.65:45701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc4cb8fb2e5d457cb5dabaf1649babc7: Network error: Client connection negotiation failed: client connection to 127.4.200.65:45701: connect: Connection refused (error 111)
I20260929 16:20:50.181234 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9841
I20260929 16:20:50.186558 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 9488
2026-09-29T16:20:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (952 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:20:50.207492 4897 mini_oidc.cc:152] Starting JWKS server
I20260929 16:20:50.209053 4897 webserver.cc:460] Webserver started at https://127.0.0.1:38259/ using document root <none> and password file <none>
I20260929 16:20:50.209210 4897 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:20:50.209438 4897 webserver.cc:460] Webserver started at http://127.0.0.1:36785/ using document root <none> and password file <none>
2026-09-29T16:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:50Z Disabled control of system clock
I20260929 16:20:50.218075 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45423
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/testchainca.cert with env {}
W20260929 16:20:50.325887 10017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:50.326172 10017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:50.326246 10017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:50.330284 10017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:50.330368 10017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:50.330442 10017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:50.330487 10017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:50.330520 10017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:20:50.330554 10017 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:20:50.330582 10017 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38259/jwks.json
I20260929 16:20:50.335914 10017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:50.336966 10017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:50.338100 10017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:50.346434 10022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.346510 10025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.346452 10023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:50.346621 10017 server_base.cc:1034] running on GCE node
I20260929 16:20:50.347349 10017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:50.348428 10017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:50.349591 10017 hybrid_clock.cc:648] HybridClock initialized: now 1790698850349559 us; error 45 us; skew 500 ppm
I20260929 16:20:50.351509 10017 webserver.cc:460] Webserver started at http://127.4.200.126:39749/ using document root <none> and password file <none>
I20260929 16:20:50.352098 10017 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:50.352162 10017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:50.352404 10017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:50.354514 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "c85dd2c2462b45cc870748fdf87ad7a0"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "a56b7d2bac8fc20a8971892e41309abf"
server_key_iv: "19f8810d29e9aad1cafbdf7dd69a08de"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.355099 10017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "c85dd2c2462b45cc870748fdf87ad7a0"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "a56b7d2bac8fc20a8971892e41309abf"
server_key_iv: "19f8810d29e9aad1cafbdf7dd69a08de"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.358800 10017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:20:50.361236 10031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:50.362439 10017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:50.362581 10017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "c85dd2c2462b45cc870748fdf87ad7a0"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "a56b7d2bac8fc20a8971892e41309abf"
server_key_iv: "19f8810d29e9aad1cafbdf7dd69a08de"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.362691 10017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:50.374747 10017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:50.375409 10017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:50.375630 10017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:50.382499 10017 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45423
I20260929 16:20:50.382468 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45423 every 8 connection(s)
I20260929 16:20:50.383474 10017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:50.386622 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:50.392025 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0: Bootstrap starting.
I20260929 16:20:50.393573 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10017
I20260929 16:20:50.393682 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:50.393982 4897 external_mini_cluster.cc:1428] Setting key 8f41570186a5e820a35ba3046b1ab095
I20260929 16:20:50.394665 10084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:50.395586 10084 log.cc:824] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:50.397450 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0: No bootstrap required, opened a new log
I20260929 16:20:50.400892 10084 raft_consensus.cc:348] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } }
I20260929 16:20:50.401146 10084 raft_consensus.cc:374] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:50.401227 10084 raft_consensus.cc:729] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85dd2c2462b45cc870748fdf87ad7a0, State: Initialized, Role: FOLLOWER
I20260929 16:20:50.401698 10084 consensus_queue.cc:260] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } }
I20260929 16:20:50.401851 10084 raft_consensus.cc:388] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:50.401916 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:50.402027 10084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:50.402918 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.395113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.4.200.126:45423)
0929 16:20:50.395771 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.395796 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.395832 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:20:50.396454 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.396479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.396795 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.397127 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.397537 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.398633 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.399515 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.399973 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.400140 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.400999 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.401004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.401016 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.401500 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.401558 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.401578 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.401685 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.401801 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.402079 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260929 16:20:50.403060 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } }
I20260929 16:20:50.403383 10084 leader_election.cc:302] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85dd2c2462b45cc870748fdf87ad7a0; no voters:
I20260929 16:20:50.403661 10084 leader_election.cc:288] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:50.403756 10089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:50.403923 10089 raft_consensus.cc:686] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [term 1 LEADER]: Becoming Leader. State: Replica: c85dd2c2462b45cc870748fdf87ad7a0, State: Running, Role: LEADER
I20260929 16:20:50.404235 10089 consensus_queue.cc:237] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } }
I20260929 16:20:50.404932 10084 sys_catalog.cc:564] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:50.405480 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c85dd2c2462b45cc870748fdf87ad7a0. Latest consensus state: current_term: 1 leader_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } } }
I20260929 16:20:50.405609 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:50.405813 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85dd2c2462b45cc870748fdf87ad7a0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45423 } } }
I20260929 16:20:50.405920 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:50.406004 10094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:50.406131 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:20:50.409391 10094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:50.415077 10094 catalog_manager.cc:1347] Generated new cluster ID: 615d8e239d3e4ed192da8dc57d32fcc6
I20260929 16:20:50.415175 10094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:50.427784 10094 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:50.428681 10094 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:50.435159 10094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c85dd2c2462b45cc870748fdf87ad7a0: Generated new TSK 0
I20260929 16:20:50.435881 10094 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:20:50.467206 10114 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:20:50.467325 10114 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:20:50.467882 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.449742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.4.200.126:45423)
0929 16:20:50.449886 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.449891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.450084 (+ 193us) server_negotiation.cc:407] Connection header received
0929 16:20:50.450277 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.450280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.450338 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.450482 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:20:50.450781 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.451672 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.452567 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.452918 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.453022 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.453364 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:20:50.467449 (+ 14085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:20:50.467734 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:20:50.467971 10087 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 16:20:50.469089 10113 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:50.449609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45423 (local address 127.0.0.1:49690)
0929 16:20:50.449993 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.450007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.450116 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.450534 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.450537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.450545 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:20:50.450664 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:50.450670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.451801 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.451804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:50.452448 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:50.452457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.453038 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.453041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:50.453111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.453159 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:20:50.467706 (+ 14547us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:20:50.467922 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.200.126:45423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.200.126:45423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:20:50.470016 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10017
2026-09-29T16:20:50Z chronyd exiting
I20260929 16:20:50.507638 4897 test_util.cc:175] -----------------------------------------------
I20260929 16:20:50.507746 4897 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790698821455595-4897-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:20:50.508797 4897 mini_oidc.cc:152] Starting JWKS server
I20260929 16:20:50.509672 4897 webserver.cc:460] Webserver started at https://127.0.0.1:42095/ using document root <none> and password file <none>
I20260929 16:20:50.509802 4897 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:20:50.510071 4897 webserver.cc:460] Webserver started at http://127.0.0.1:42463/ using document root <none> and password file <none>
2026-09-29T16:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:50Z Disabled control of system clock
I20260929 16:20:50.518760 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:46089
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:46089
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/testchainca.cert with env {}
W20260929 16:20:50.627812 10120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:50.628064 10120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:50.628103 10120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:50.632082 10120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:50.632135 10120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:50.632150 10120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:50.632166 10120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:50.632181 10120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:20:50.632197 10120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:20:50.632210 10120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42095/jwks.json
I20260929 16:20:50.637449 10120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:46089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:50.638648 10120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:50.639614 10120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:50.648110 10120 server_base.cc:1034] running on GCE node
W20260929 16:20:50.648049 10126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.648029 10128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.648020 10125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:50.648799 10120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:50.649890 10120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:50.651078 10120 hybrid_clock.cc:648] HybridClock initialized: now 1790698850651065 us; error 25 us; skew 500 ppm
I20260929 16:20:50.652956 10120 webserver.cc:460] Webserver started at http://127.4.200.126:43079/ using document root <none> and password file <none>
I20260929 16:20:50.653587 10120 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:50.653676 10120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:50.653899 10120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:50.656003 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "08bcfbeb48624f3084a6d081ec640be4"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e766176f0014197c263a9e005f1f17c1"
server_key_iv: "ccdf0d0af192ea2ad25aba2a42c9c258"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.656586 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "08bcfbeb48624f3084a6d081ec640be4"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e766176f0014197c263a9e005f1f17c1"
server_key_iv: "ccdf0d0af192ea2ad25aba2a42c9c258"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.660238 10120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:20:50.662663 10134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:50.664052 10120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:50.664214 10120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "08bcfbeb48624f3084a6d081ec640be4"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e766176f0014197c263a9e005f1f17c1"
server_key_iv: "ccdf0d0af192ea2ad25aba2a42c9c258"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.664337 10120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:50.682004 10120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:50.682819 10120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:50.683034 10120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:50.689991 10120 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:46089
I20260929 16:20:50.690017 10186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:46089 every 8 connection(s)
I20260929 16:20:50.691020 10120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:50.694234 10187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:50.694228 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10120
I20260929 16:20:50.694401 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:50.694713 4897 external_mini_cluster.cc:1428] Setting key cd4c3d452a3e33560c10b42a75353deb
I20260929 16:20:50.701553 10187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4: Bootstrap starting.
I20260929 16:20:50.704542 10187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:50.705412 10187 log.cc:824] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:50.705476 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.696146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.4.200.126:46089)
0929 16:20:50.696828 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.696847 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.696893 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:20:50.697710 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.697734 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.698258 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.698652 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:50.699095 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.700564 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.701496 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.702160 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.702319 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.703314 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:50.703320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:50.703336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:50.703840 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:50.703911 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:50.703938 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:50.704047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:50.704207 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 16:20:50.704496 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260929 16:20:50.707755 10187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4: No bootstrap required, opened a new log
I20260929 16:20:50.708526 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:20:50.712013 10187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } }
I20260929 16:20:50.712248 10187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:50.712282 10187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bcfbeb48624f3084a6d081ec640be4, State: Initialized, Role: FOLLOWER
I20260929 16:20:50.712651 10187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } }
I20260929 16:20:50.712764 10187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:50.712834 10187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:50.712936 10187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:50.713963 10187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } }
I20260929 16:20:50.714329 10187 leader_election.cc:302] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08bcfbeb48624f3084a6d081ec640be4; no voters:
I20260929 16:20:50.714731 10187 leader_election.cc:288] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:50.714898 10192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:50.715142 10192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [term 1 LEADER]: Becoming Leader. State: Replica: 08bcfbeb48624f3084a6d081ec640be4, State: Running, Role: LEADER
I20260929 16:20:50.715423 10192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } }
I20260929 16:20:50.716051 10187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:50.717274 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08bcfbeb48624f3084a6d081ec640be4. Latest consensus state: current_term: 1 leader_uuid: "08bcfbeb48624f3084a6d081ec640be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } } }
I20260929 16:20:50.717324 10193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08bcfbeb48624f3084a6d081ec640be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bcfbeb48624f3084a6d081ec640be4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 46089 } } }
I20260929 16:20:50.717415 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:50.717415 10193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:50.717839 10201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:50.720671 10201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:50.726267 10201 catalog_manager.cc:1347] Generated new cluster ID: a993021910c247dda669ae72933130d1
I20260929 16:20:50.726357 10201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:50.764599 10201 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:50.765784 10201 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:50.773770 10201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08bcfbeb48624f3084a6d081ec640be4: Generated new TSK 0
I20260929 16:20:50.774616 10201 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:20:50.800237 10217 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:20:50.800325 10217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:20:50.800911 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:50.785280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.4.200.126:46089)
0929 16:20:50.785407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:50.785411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:50.785609 (+ 198us) server_negotiation.cc:407] Connection header received
0929 16:20:50.785765 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:50.785768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:50.785813 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:50.785900 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:20:50.786167 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.786725 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.787953 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.788180 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.788271 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.788581 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:20:50.800606 (+ 12025us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:20:50.800806 (+ 200us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:20:50.801000 10190 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:20:50.801126 10216 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:50.785145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:46089 (local address 127.0.0.1:40680)
0929 16:20:50.785497 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:50.785516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:50.785642 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:50.785930 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:50.785933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:50.785942 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:20:50.786060 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:50.786067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.786839 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.786842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:50.787814 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:50.787824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:50.788300 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:50.788304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:50.788397 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:50.788423 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:20:50.800779 (+ 12356us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:20:50.800942 (+ 163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.200.126:46089: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260929 16:20:50.801704 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10120
2026-09-29T16:20:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:20:50.840340 4897 mini_oidc.cc:152] Starting JWKS server
I20260929 16:20:50.841162 4897 webserver.cc:460] Webserver started at https://127.0.0.1:45109/ using document root <none> and password file <none>
I20260929 16:20:50.841240 4897 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:20:50.841490 4897 webserver.cc:460] Webserver started at http://127.0.0.1:36451/ using document root <none> and password file <none>
2026-09-29T16:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:50Z Disabled control of system clock
I20260929 16:20:50.849889 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45393
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45109/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:20:50.957340 10223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:50.957662 10223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:50.957741 10223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:50.961866 10223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:50.961972 10223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:50.962040 10223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:50.962095 10223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:50.962165 10223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:20:50.962212 10223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:20:50.962245 10223 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45109/jwks.json
I20260929 16:20:50.967223 10223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45109/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:50.968348 10223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:50.969746 10223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:50.978538 10223 server_base.cc:1034] running on GCE node
W20260929 16:20:50.978499 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.978504 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:50.978494 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:50.979296 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:50.980394 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:50.981557 10223 hybrid_clock.cc:648] HybridClock initialized: now 1790698850981546 us; error 28 us; skew 500 ppm
I20260929 16:20:50.983567 10223 webserver.cc:460] Webserver started at http://127.4.200.126:40651/ using document root <none> and password file <none>
I20260929 16:20:50.984176 10223 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:50.984239 10223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:50.984465 10223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:50.986630 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "5793530153b34e0c9d598696b729ca6a"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e5484336dc2f167053f1026e12f63486"
server_key_iv: "983b48e6fd4e9ac7c11baa4f4a4230b8"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.987200 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "5793530153b34e0c9d598696b729ca6a"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e5484336dc2f167053f1026e12f63486"
server_key_iv: "983b48e6fd4e9ac7c11baa4f4a4230b8"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.990868 10223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:50.993201 10237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:50.994503 10223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:50.994633 10223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "5793530153b34e0c9d598696b729ca6a"
format_stamp: "Formatted at 2026-09-29 16:20:50 on dist-test-slave-s9ct"
server_key: "e5484336dc2f167053f1026e12f63486"
server_key_iv: "983b48e6fd4e9ac7c11baa4f4a4230b8"
server_key_version: "encryptionkey@0"
I20260929 16:20:50.994747 10223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:51.022847 10223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:51.023590 10223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:51.023801 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:51.030862 10223 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45393
I20260929 16:20:51.030881 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45393 every 8 connection(s)
I20260929 16:20:51.031839 10223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:51.034715 10290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:51.035645 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10223
I20260929 16:20:51.035789 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:51.036073 4897 external_mini_cluster.cc:1428] Setting key cf62691cf6053c5a79db284438dc1eac
I20260929 16:20:51.041615 10290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a: Bootstrap starting.
I20260929 16:20:51.044284 10290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:51.045140 10290 log.cc:824] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:51.046679 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:51.037470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.4.200.126:45393)
0929 16:20:51.038174 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:51.038194 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:51.038237 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:20:51.039093 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:51.039117 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:51.039633 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:51.040003 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:51.040460 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.041908 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.043001 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.043688 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.043844 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.044775 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:51.044780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:51.044793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:51.045191 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:51.045245 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:51.045293 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:51.045387 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:51.045574 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 16:20:51.045854 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260929 16:20:51.047351 10290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a: No bootstrap required, opened a new log
I20260929 16:20:51.049489 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:20:51.051350 10290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } }
I20260929 16:20:51.051563 10290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:51.051625 10290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5793530153b34e0c9d598696b729ca6a, State: Initialized, Role: FOLLOWER
I20260929 16:20:51.052042 10290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } }
I20260929 16:20:51.052165 10290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:51.052248 10290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:51.052356 10290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:51.053285 10290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } }
I20260929 16:20:51.053625 10290 leader_election.cc:302] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5793530153b34e0c9d598696b729ca6a; no voters:
I20260929 16:20:51.053972 10290 leader_election.cc:288] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:51.054121 10295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:51.054371 10295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [term 1 LEADER]: Becoming Leader. State: Replica: 5793530153b34e0c9d598696b729ca6a, State: Running, Role: LEADER
I20260929 16:20:51.054999 10295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } }
I20260929 16:20:51.055371 10290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:51.056572 10296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5793530153b34e0c9d598696b729ca6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } } }
I20260929 16:20:51.056699 10296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:51.057057 10297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5793530153b34e0c9d598696b729ca6a. Latest consensus state: current_term: 1 leader_uuid: "5793530153b34e0c9d598696b729ca6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5793530153b34e0c9d598696b729ca6a" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45393 } } }
I20260929 16:20:51.057156 10297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:51.057238 10304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:51.059991 10304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:51.065152 10304 catalog_manager.cc:1347] Generated new cluster ID: 96c8f83e1bdc45119aa76699bb22efc1
I20260929 16:20:51.065220 10304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:51.089469 10304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:51.090566 10304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:51.094962 10304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5793530153b34e0c9d598696b729ca6a: Generated new TSK 0
I20260929 16:20:51.095630 10304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:20:51.113518 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:51.108211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.4.200.126:45393)
0929 16:20:51.108339 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:51.108343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:51.108504 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:20:51.108659 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:51.108663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:51.108723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:51.108841 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:20:51.109124 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.109939 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.110910 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.111264 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.111368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.111713 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:20:51.113154 (+ 1441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:20:51.113376 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 16:20:51.113587 10293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:20:51.115211 10319 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:51.108097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45393 (local address 127.0.0.1:52618)
0929 16:20:51.108405 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:51.108419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:51.108524 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:51.108870 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:51.108873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:51.108894 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:20:51.108998 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:51.109004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.110079 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.110082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:51.110789 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:51.110799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.111409 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.111413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:51.111500 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.111522 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:20:51.113312 (+ 1790us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:20:51.115042 (+ 1730us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.200.126:45393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1535,"thread_start_us":110,"threads_started":1}
I20260929 16:20:51.115741 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10223
E20260929 16:20:51.123901 10320 webserver.cc:533] Webserver: sslize failed:
E20260929 16:20:51.123962 10320 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T16:20:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:51Z Disabled control of system clock
I20260929 16:20:51.162891 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:32773
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:32773
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/testchainca.cert with env {}
W20260929 16:20:51.272061 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:51.272300 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:51.272339 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:51.276257 10324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:51.276309 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:51.276328 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:51.276348 10324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:51.276366 10324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:20:51.276393 10324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:20:51.276417 10324 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:20:51.281579 10324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:32773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:32773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:51.282611 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:51.283545 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:51.291790 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:51.291780 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:51.291780 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:51.292040 10324 server_base.cc:1034] running on GCE node
I20260929 16:20:51.292641 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:51.293660 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:51.294847 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790698851294817 us; error 49 us; skew 500 ppm
I20260929 16:20:51.296664 10324 webserver.cc:460] Webserver started at http://127.4.200.126:36203/ using document root <none> and password file <none>
I20260929 16:20:51.297238 10324 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:51.297334 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:51.297592 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:51.299621 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "61e38363c3ea4252b8d693a310abff83"
format_stamp: "Formatted at 2026-09-29 16:20:51 on dist-test-slave-s9ct"
server_key: "ceb8d23287c5df118dcd557ddef9933c"
server_key_iv: "78a716df3872d299ee9bbb26d5f0ed1e"
server_key_version: "encryptionkey@0"
I20260929 16:20:51.300135 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "61e38363c3ea4252b8d693a310abff83"
format_stamp: "Formatted at 2026-09-29 16:20:51 on dist-test-slave-s9ct"
server_key: "ceb8d23287c5df118dcd557ddef9933c"
server_key_iv: "78a716df3872d299ee9bbb26d5f0ed1e"
server_key_version: "encryptionkey@0"
I20260929 16:20:51.303546 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:20:51.305850 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:51.307062 10324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:51.307160 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "61e38363c3ea4252b8d693a310abff83"
format_stamp: "Formatted at 2026-09-29 16:20:51 on dist-test-slave-s9ct"
server_key: "ceb8d23287c5df118dcd557ddef9933c"
server_key_iv: "78a716df3872d299ee9bbb26d5f0ed1e"
server_key_version: "encryptionkey@0"
I20260929 16:20:51.307245 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:51.325645 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:51.326210 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:51.326346 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:51.332998 10324 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:32773
I20260929 16:20:51.333021 10390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:32773 every 8 connection(s)
I20260929 16:20:51.333909 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:51.336838 10391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:51.338454 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10324
I20260929 16:20:51.338588 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:51.338860 4897 external_mini_cluster.cc:1428] Setting key e492f818adeff53ba7e77f57f4d3b916
I20260929 16:20:51.343098 10391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83: Bootstrap starting.
I20260929 16:20:51.345258 10391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:51.346091 10391 log.cc:824] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:51.347862 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:51.340214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.4.200.126:32773)
0929 16:20:51.340839 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:51.340853 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:51.340887 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:20:51.341509 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:51.341525 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:51.341911 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:51.342231 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:51.342678 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.343672 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.344540 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.344996 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.345159 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.346091 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:51.346096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:51.346108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:20:51.346604 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:51.346661 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:20:51.346682 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:51.346788 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:51.346917 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 16:20:51.347167 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260929 16:20:51.348053 10391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83: No bootstrap required, opened a new log
I20260929 16:20:51.350651 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:20:51.352339 10391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } }
I20260929 16:20:51.352618 10391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:51.352697 10391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e38363c3ea4252b8d693a310abff83, State: Initialized, Role: FOLLOWER
I20260929 16:20:51.353092 10391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } }
I20260929 16:20:51.353236 10391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:51.353323 10391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:51.353421 10391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:51.354318 10391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } }
I20260929 16:20:51.354736 10391 leader_election.cc:302] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61e38363c3ea4252b8d693a310abff83; no voters:
I20260929 16:20:51.355074 10391 leader_election.cc:288] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:51.355196 10396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:51.355453 10396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [term 1 LEADER]: Becoming Leader. State: Replica: 61e38363c3ea4252b8d693a310abff83, State: Running, Role: LEADER
I20260929 16:20:51.355768 10396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } }
I20260929 16:20:51.356350 10391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:51.357471 10398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e38363c3ea4252b8d693a310abff83. Latest consensus state: current_term: 1 leader_uuid: "61e38363c3ea4252b8d693a310abff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } } }
I20260929 16:20:51.357610 10398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:51.357870 10397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61e38363c3ea4252b8d693a310abff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e38363c3ea4252b8d693a310abff83" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32773 } } }
I20260929 16:20:51.357959 10397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:51.358093 10405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:51.361361 10405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:51.366989 10405 catalog_manager.cc:1347] Generated new cluster ID: 4038bf2eb73d4b67b8f7b8da85fbdc87
I20260929 16:20:51.367060 10405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:51.389289 10405 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:51.390255 10405 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:51.395044 10405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61e38363c3ea4252b8d693a310abff83: Generated new TSK 0
I20260929 16:20:51.395817 10405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:20:51.454639 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:51.409287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.4.200.126:32773)
0929 16:20:51.409415 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:51.409420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:51.409596 (+ 176us) server_negotiation.cc:407] Connection header received
0929 16:20:51.409772 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:51.409775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:51.409823 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:51.409933 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:20:51.410195 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.410852 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.411738 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.412011 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.412125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.412440 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:20:51.454080 (+ 41640us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:20:51.454453 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:20:51.454733 10394 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 16:20:51.454806 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:51.409211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:32773 (local address 127.0.0.1:57194)
0929 16:20:51.409500 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:51.409512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:51.409636 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:51.409943 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:51.409947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:51.409955 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:20:51.410085 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:51.410091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.410963 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.410968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:51.411615 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:51.411622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:51.412133 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:51.412138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:51.412230 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:51.412261 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:20:51.454324 (+ 42063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:20:51.454608 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.200.126:32773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260929 16:20:51.455240 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10324
2026-09-29T16:20:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (319 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:51 dist-test-slave-s9ct krb5kdc[10432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:51 dist-test-slave-s9ct krb5kdc[10432](info): set up 2 sockets
Sep 29 16:20:51 dist-test-slave-s9ct krb5kdc[10432](info): commencing operation
krb5kdc: starting...
W20260929 16:20:52.782846 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:52 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:20:52.903575 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:37545
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:53.012727 10445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:53.012995 10445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:53.013033 10445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:53.016899 10445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:53.016950 10445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:53.016970 10445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:53.016987 10445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:53.017002 10445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:53.022176 10445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:53.023226 10445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:53.024194 10445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:53.032240 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.032361 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.032274 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:53.032646 10445 server_base.cc:1034] running on GCE node
I20260929 16:20:53.033223 10445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:53.034264 10445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:53.035455 10445 hybrid_clock.cc:648] HybridClock initialized: now 1790698853035440 us; error 36 us; skew 500 ppm
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.038148 10445 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:20:53.039112 10445 webserver.cc:460] Webserver started at http://127.4.200.126:34093/ using document root <none> and password file <none>
I20260929 16:20:53.039669 10445 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:53.039757 10445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:53.039986 10445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:53.041983 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "5e4f1dd231934fec88854b02a8950a79"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "2440d5a2ef18bda9b8c1c05a37e7308d"
server_key_iv: "63b077bb55ec9d0fa10a6bc7d4068194"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.042613 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "5e4f1dd231934fec88854b02a8950a79"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "2440d5a2ef18bda9b8c1c05a37e7308d"
server_key_iv: "63b077bb55ec9d0fa10a6bc7d4068194"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.046343 10445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:20:53.048781 10460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.050031 10445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:53.050176 10445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "5e4f1dd231934fec88854b02a8950a79"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "2440d5a2ef18bda9b8c1c05a37e7308d"
server_key_iv: "63b077bb55ec9d0fa10a6bc7d4068194"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.050292 10445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:53.063032 10445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:53.066653 10445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:53.066861 10445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:53.074124 10445 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:37545
I20260929 16:20:53.074155 10512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:37545 every 8 connection(s)
I20260929 16:20:53.075148 10445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:53.077823 10513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:53.079666 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10445
I20260929 16:20:53.079804 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:53.080068 4897 external_mini_cluster.cc:1428] Setting key 0e6aff88c532978392ebea701dcd1aa7
I20260929 16:20:53.084769 10513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79: Bootstrap starting.
I20260929 16:20:53.087486 10513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:53.088231 10513 log.cc:824] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:53.090590 10513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79: No bootstrap required, opened a new log
I20260929 16:20:53.094592 10513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } }
I20260929 16:20:53.094825 10513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:53.094882 10513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f1dd231934fec88854b02a8950a79, State: Initialized, Role: FOLLOWER
I20260929 16:20:53.095312 10513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } }
I20260929 16:20:53.095430 10513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:53.095537 10513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:53.095631 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.081515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.4.200.126:37545)
0929 16:20:53.081945 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.081964 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.082007 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:20:53.082874 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.082898 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.083325 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.083702 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.084369 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.085796 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.086715 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.087378 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.087546 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.090560 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.090565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.090578 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.090604 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.092457 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.092972 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.092977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.092981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.093093 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.093382 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.093384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.093385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.093784 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.093949 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.094257 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.094623 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260929 16:20:53.095659 10513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:53.096642 10513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } }
I20260929 16:20:53.096990 10513 leader_election.cc:302] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e4f1dd231934fec88854b02a8950a79; no voters:
I20260929 16:20:53.097342 10513 leader_election.cc:288] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:53.097432 10518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:53.097656 10518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [term 1 LEADER]: Becoming Leader. State: Replica: 5e4f1dd231934fec88854b02a8950a79, State: Running, Role: LEADER
I20260929 16:20:53.098023 10518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } }
I20260929 16:20:53.098649 10513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:53.099224 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e4f1dd231934fec88854b02a8950a79. Latest consensus state: current_term: 1 leader_uuid: "5e4f1dd231934fec88854b02a8950a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } } }
I20260929 16:20:53.099421 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:53.099689 10519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e4f1dd231934fec88854b02a8950a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4f1dd231934fec88854b02a8950a79" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 37545 } } }
I20260929 16:20:53.099781 10519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:53.099932 10527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:53.102974 10527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:53.108603 10527 catalog_manager.cc:1347] Generated new cluster ID: a1d1d862353048fea4966dd823b608f1
I20260929 16:20:53.108698 10527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:53.126973 10527 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:53.128279 10527 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:53.134303 10527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e4f1dd231934fec88854b02a8950a79: Generated new TSK 0
I20260929 16:20:53.135020 10527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:20:53.164981 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:53.274298 10539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:53.274571 10539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:53.274616 10539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:53.278479 10539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:53.278527 10539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:53.278582 10539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:53.283633 10539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:53.284592 10539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:53.285552 10539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:20:53.295158 10539 server_base.cc:1034] running on GCE node
W20260929 16:20:53.295203 10544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.295198 10545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.295365 10547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:53.295872 10539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:53.296510 10539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:53.297700 10539 hybrid_clock.cc:648] HybridClock initialized: now 1790698853297681 us; error 42 us; skew 500 ppm
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.300699 10539 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:20:53.301558 10539 webserver.cc:460] Webserver started at http://127.4.200.65:44623/ using document root <none> and password file <none>
I20260929 16:20:53.302130 10539 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:53.302223 10539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:53.302511 10539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:53.304556 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "c83801efc1ed4947b5bbedf73461ae64"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "8e418a6cd6232d6eeedeafc98b9d2bc2"
server_key_iv: "4d6e5f4f6bc3c21259a21ca791737244"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.305109 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "c83801efc1ed4947b5bbedf73461ae64"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "8e418a6cd6232d6eeedeafc98b9d2bc2"
server_key_iv: "4d6e5f4f6bc3c21259a21ca791737244"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.309250 10539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:20:53.311695 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.312817 10539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:53.312961 10539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "c83801efc1ed4947b5bbedf73461ae64"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "8e418a6cd6232d6eeedeafc98b9d2bc2"
server_key_iv: "4d6e5f4f6bc3c21259a21ca791737244"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.313082 10539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:53.335285 10539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:53.338321 10539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:53.338568 10539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:53.339193 10539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:53.340217 10539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:53.340274 10539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.340325 10539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:53.340353 10539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.349315 10539 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:32777
I20260929 16:20:53.349332 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:32777 every 8 connection(s)
I20260929 16:20:53.350260 10539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:53.351243 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10539
I20260929 16:20:53.351375 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:53.351696 4897 external_mini_cluster.cc:1428] Setting key a46ba046fc090744c4f485e3a1b701e8
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:20:53.366873 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.352454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:46469 (local address 127.4.200.126:37545)
0929 16:20:53.352595 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.352599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.353557 (+ 958us) server_negotiation.cc:407] Connection header received
0929 16:20:53.355360 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.355365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.355431 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.355528 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.356702 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.357287 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.359256 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.359492 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.359583 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.362942 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.362946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.362948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.362977 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.364514 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.365169 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.365172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.365173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.365264 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.365675 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.365679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.365681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.365883 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.366003 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.366570 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.366705 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:20:53.367844 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.352880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:37545 (local address 127.4.200.65:46469)
0929 16:20:53.353406 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:53.353463 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:53.355117 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:53.355700 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:53.355707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:53.356071 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:53.356561 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.356574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.357466 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.357473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.359110 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.359125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.359680 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.359687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.359940 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.360695 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:53.360720 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:53.362718 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:53.364688 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.364696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.364716 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.365014 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.365375 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.365379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.365382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.365565 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.365999 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:53.366006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:53.366285 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 16:20:53.366516 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:20:53.366873 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":154,"threads_started":1}
I20260929 16:20:53.369215 10668 heartbeater.cc:347] Connected to a master server at 127.4.200.126:37545
I20260929 16:20:53.369513 10668 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:53.370193 10668 heartbeater.cc:511] Master 127.4.200.126:37545 requested a full tablet report, sending...
I20260929 16:20:53.371954 10477 ts_manager.cc:194] Registered new tserver with Master: c83801efc1ed4947b5bbedf73461ae64 (127.4.200.65:32777)
I20260929 16:20:53.373196 10477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:46469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:20:53.382330 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:53.499502 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:53.499788 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:53.499833 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:53.504204 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:53.504269 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:53.504364 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:53.509788 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:53.510955 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:53.511989 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:53.521471 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.521471 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:53.521631 10673 server_base.cc:1034] running on GCE node
W20260929 16:20:53.521534 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:53.522202 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:53.522933 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:53.524118 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790698853524102 us; error 30 us; skew 500 ppm
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.526911 10673 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:20:53.527709 10673 webserver.cc:460] Webserver started at http://127.4.200.66:40465/ using document root <none> and password file <none>
I20260929 16:20:53.528211 10673 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:53.528266 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:53.528427 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:53.530380 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "4bd57198f385473da5853d79d83f000c"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "3fc742e927a622464389d91404e1c4ec"
server_key_iv: "6dcd1f5114829b2b7c2f43cab434ab93"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.531023 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "4bd57198f385473da5853d79d83f000c"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "3fc742e927a622464389d91404e1c4ec"
server_key_iv: "6dcd1f5114829b2b7c2f43cab434ab93"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.534749 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:20:53.537088 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.538270 10673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:53.538363 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "4bd57198f385473da5853d79d83f000c"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "3fc742e927a622464389d91404e1c4ec"
server_key_iv: "6dcd1f5114829b2b7c2f43cab434ab93"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.538519 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:53.557809 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:53.561141 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:53.561295 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:53.561831 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:53.562660 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:53.562709 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.562749 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:53.562765 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.572906 10673 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:34331
I20260929 16:20:53.572927 10801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:34331 every 8 connection(s)
I20260929 16:20:53.573839 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:53.578775 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10673
I20260929 16:20:53.578933 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:20:53.579241 4897 external_mini_cluster.cc:1428] Setting key 15ed68c30d8c086c69a3f33e2ecbeec6
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:53.589330 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.575736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:58923 (local address 127.4.200.126:37545)
0929 16:20:53.575890 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.575894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.576782 (+ 888us) server_negotiation.cc:407] Connection header received
0929 16:20:53.578110 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.578112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.578157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.578232 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.579620 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.580186 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.581533 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.581971 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.582104 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.585160 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.585167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.585173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.585209 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.586889 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.587548 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.587551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.587552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.587625 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.588020 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.588022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.588024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.588198 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.588274 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.588981 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.589126 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:20:53.590504 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.576180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:37545 (local address 127.4.200.66:58923)
0929 16:20:53.576627 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:53.576670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:53.577907 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:53.578450 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:53.578460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:53.578835 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:53.579462 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.579478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.580318 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.580322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.581389 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.581398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.582122 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.582124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.582278 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.582967 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:53.582994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:53.584966 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:53.587015 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.587024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.587045 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.587446 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.587716 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.587722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.587727 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.587923 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.588310 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:53.588319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:53.588682 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 16:20:53.588923 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 16:20:53.589333 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260929 16:20:53.591663 10802 heartbeater.cc:347] Connected to a master server at 127.4.200.126:37545
I20260929 16:20:53.591954 10802 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:20:53.592548 10802 heartbeater.cc:511] Master 127.4.200.126:37545 requested a full tablet report, sending...
I20260929 16:20:53.593664 10477 ts_manager.cc:194] Registered new tserver with Master: 4bd57198f385473da5853d79d83f000c (127.4.200.66:34331)
I20260929 16:20:53.594244 10477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:58923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:20:53.609105 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:53.720534 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:53.720782 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:53.720824 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:53.724921 10807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:53.724972 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:53.725061 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:53.730197 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:53.731242 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:53.732218 10807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:53.741489 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.741456 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:53.741478 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:53.741741 10807 server_base.cc:1034] running on GCE node
I20260929 16:20:53.742153 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:53.742826 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:53.743979 10807 hybrid_clock.cc:648] HybridClock initialized: now 1790698853743950 us; error 41 us; skew 500 ppm
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.747179 10807 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:20:53.747967 10807 webserver.cc:460] Webserver started at http://127.4.200.67:33537/ using document root <none> and password file <none>
I20260929 16:20:53.748497 10807 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:53.748594 10807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:53.748818 10807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:53.750849 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "38f8571f8e094296b173f297ebae21a6"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "765a0805350deced723626a4a8dc8b8c"
server_key_iv: "c1cf2d7910b128bddf5e67fe222cda60"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.751380 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "38f8571f8e094296b173f297ebae21a6"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "765a0805350deced723626a4a8dc8b8c"
server_key_iv: "c1cf2d7910b128bddf5e67fe222cda60"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.755116 10807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:20:53.757404 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.758582 10807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 16:20:53.758693 10807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "38f8571f8e094296b173f297ebae21a6"
format_stamp: "Formatted at 2026-09-29 16:20:53 on dist-test-slave-s9ct"
server_key: "765a0805350deced723626a4a8dc8b8c"
server_key_iv: "c1cf2d7910b128bddf5e67fe222cda60"
server_key_version: "encryptionkey@0"
I20260929 16:20:53.758807 10807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:53.770761 10807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:53.773478 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:53.773664 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:53.774237 10807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:53.775061 10807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:53.775107 10807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.775173 10807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:53.775205 10807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:53.783541 10807 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:34989
I20260929 16:20:53.783577 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:34989 every 8 connection(s)
I20260929 16:20:53.784451 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:53.785157 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10807
I20260929 16:20:53.785313 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:53.785622 4897 external_mini_cluster.cc:1428] Setting key 5c70222f1f27c6c7581c0c8e82f6a1a6
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:20:53.797767 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.786326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:45115 (local address 127.4.200.126:37545)
0929 16:20:53.786494 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.786498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.787512 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 16:20:53.788993 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.788996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.789039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.789109 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.790189 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.790731 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.791828 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.792069 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.792170 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.794580 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.794583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.794586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.794611 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.795943 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.796442 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.796445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.796446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.796524 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.796814 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.796816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.796817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.797005 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.797086 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.797527 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.797636 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:53.798879 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.786812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:37545 (local address 127.4.200.67:45115)
0929 16:20:53.787367 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:53.787423 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:53.788792 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:53.789241 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:53.789248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:53.789593 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:53.790067 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.790079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.790829 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.790832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.791682 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:53.791690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.792207 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.792209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:53.792345 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.792922 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:53.792938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:53.794415 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:53.796061 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.796065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.796077 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.796330 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.796604 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:53.796606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:53.796607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:53.796732 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:53.797097 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:53.797102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:53.797321 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 16:20:53.797475 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:20:53.797727 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":166,"threads_started":1}
I20260929 16:20:53.800149 10936 heartbeater.cc:347] Connected to a master server at 127.4.200.126:37545
I20260929 16:20:53.800418 10936 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:53.800982 10936 heartbeater.cc:511] Master 127.4.200.126:37545 requested a full tablet report, sending...
I20260929 16:20:53.801994 10477 ts_manager.cc:194] Registered new tserver with Master: 38f8571f8e094296b173f297ebae21a6 (127.4.200.67:34989)
I20260929 16:20:53.802621 10477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:45115
I20260929 16:20:53.811882 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/test-admin.keytab.
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.827265 4897 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:20:53.829937 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:53.828531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.4.200.65:32777)
0929 16:20:53.828831 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.828845 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.828869 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:20:53.828918 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.828923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.829106 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.829224 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.829728 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46818: BlockingRecv error: recv got EOF from 127.0.0.1:46818 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
W20260929 16:20:53.830030 4897 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.831080 4897 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.831976 4897 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:53.842502 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.833319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.4.200.65:32777)
0929 16:20:53.833470 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.833474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.833551 (+ 77us) server_negotiation.cc:407] Connection header received
0929 16:20:53.833755 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.833758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.833799 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.833884 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.834443 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.835218 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.836074 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.836465 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.836583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.838971 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.838980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.838993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.839018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.840528 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.840994 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.840999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.841000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.841077 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.841367 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.841370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.841372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.841736 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.841947 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.842140 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.842295 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:20:53.846448 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:53.845544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.4.200.65:32777)
0929 16:20:53.845690 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.845693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.845708 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:20:53.845817 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.845819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.845852 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.845914 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.846313 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46830: BlockingRecv error: recv got EOF from 127.0.0.1:46830 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:20:53.846485 4897 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.847496 4897 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.848357 4897 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:53.857003 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.849116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.4.200.65:32777)
0929 16:20:53.849261 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.849264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.849278 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:53.849381 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.849383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.849416 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.849481 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.849919 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.850501 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.851269 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.851480 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.851558 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.853619 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.853622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.853625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.853668 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.855348 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.855773 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.855776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.855778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.855865 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.856139 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.856145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.856150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.856328 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.856462 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.856714 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.856837 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:20:53.859568 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:20:53.858724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.4.200.65:32777)
0929 16:20:53.858861 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.858864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.858913 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:20:53.859005 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.859018 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.859055 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.859118 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.859455 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46856: BlockingRecv error: recv got EOF from 127.0.0.1:46856 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 16:20:53.859915 4897 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790698821455595-4897-0/krb5kdc/krb5cc)
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.860948 4897 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:53.861812 4897 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:20:53.869916 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:53.862688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.4.200.65:32777)
0929 16:20:53.862817 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:53.862821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:53.862864 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:20:53.862957 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:53.862958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:53.862992 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:53.863060 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:53.863555 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.864041 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.864790 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:53.864983 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:53.865064 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:53.867158 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:53.867161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:53.867163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:53.867183 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:53.868413 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.868784 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.868787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.868788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.868855 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:53.869106 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:53.869109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:53.869111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:53.869285 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:53.869406 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:53.869585 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:20:53.869751 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:20:53.870983 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10539
I20260929 16:20:53.876533 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10673
I20260929 16:20:53.882019 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10807
I20260929 16:20:53.887494 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10445
2026-09-29T16:20:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2178 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10956](info): set up 2 sockets
Sep 29 16:20:53 dist-test-slave-s9ct krb5kdc[10956](info): commencing operation
krb5kdc: starting...
W20260929 16:20:54.926964 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:20:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab.
Entry for principal oryx/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab.
I20260929 16:20:55.047564 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33163
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab
--principal=oryx/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:55.161360 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:55.161640 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:55.161685 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:55.166368 10969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:55.166486 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:55.166554 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:55.166592 10969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:55.166628 10969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:55.172591 10969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab
--principal=oryx/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:55.173686 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:55.174791 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:55.183236 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.183267 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.183238 10977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:55.183508 10969 server_base.cc:1034] running on GCE node
I20260929 16:20:55.184201 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:55.185324 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:55.186487 10969 hybrid_clock.cc:648] HybridClock initialized: now 1790698855186461 us; error 33 us; skew 500 ppm
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:55.189311 10969 init.cc:377] Logged in from keytab as oryx/127.4.200.126@KRBTEST.COM (short username oryx)
I20260929 16:20:55.190258 10969 webserver.cc:460] Webserver started at http://127.4.200.126:34441/ using document root <none> and password file <none>
I20260929 16:20:55.190889 10969 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:55.190979 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:55.191187 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:55.193205 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "128bfa733ff744bbbe6e105ec0ad0a62"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "571c126e949395ef41c200e5d1689930"
server_key_iv: "ec5ae9d7dca7cb9f90ecbcf1325ec516"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.193770 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "128bfa733ff744bbbe6e105ec0ad0a62"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "571c126e949395ef41c200e5d1689930"
server_key_iv: "ec5ae9d7dca7cb9f90ecbcf1325ec516"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.197540 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:20:55.200088 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.201277 10969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:55.201419 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "128bfa733ff744bbbe6e105ec0ad0a62"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "571c126e949395ef41c200e5d1689930"
server_key_iv: "ec5ae9d7dca7cb9f90ecbcf1325ec516"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.201552 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:55.224999 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:55.228380 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:55.228653 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:55.235996 10969 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:33163
I20260929 16:20:55.236311 11036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:33163 every 8 connection(s)
I20260929 16:20:55.237011 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:20:55.239732 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:55.244066 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 10969
I20260929 16:20:55.244222 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:20:55.244526 4897 external_mini_cluster.cc:1428] Setting key 7d363844beb9bfc56be82acffb42b31a
I20260929 16:20:55.245625 11037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Bootstrap starting.
I20260929 16:20:55.247975 11037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:55.248888 11037 log.cc:824] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:55.251116 11037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: No bootstrap required, opened a new log
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:55.256029 11037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } }
I20260929 16:20:55.256342 11037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:55.256394 11037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128bfa733ff744bbbe6e105ec0ad0a62, State: Initialized, Role: FOLLOWER
I20260929 16:20:55.256918 11037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } }
I20260929 16:20:55.257105 11037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:20:55.257180 11037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:20:55.257282 11037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:55.258520 11037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } }
I20260929 16:20:55.258880 11037 leader_election.cc:302] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128bfa733ff744bbbe6e105ec0ad0a62; no voters:
I20260929 16:20:55.259297 11037 leader_election.cc:288] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:20:55.259459 11042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:55.259795 11042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 LEADER]: Becoming Leader. State: Replica: 128bfa733ff744bbbe6e105ec0ad0a62, State: Running, Role: LEADER
I20260929 16:20:55.260218 11042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } }
I20260929 16:20:55.260815 11037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:55.261310 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.245974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.4.200.126:33163)
0929 16:20:55.246427 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:55.246445 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:55.246494 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:20:55.247111 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:55.247128 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:55.247448 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:55.247734 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:55.248730 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.250028 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.251230 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.251742 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.251866 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.254711 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:55.254718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:55.254738 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:55.254772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:55.256768 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.257343 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.257349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.257356 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.257510 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.257850 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.257853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.257855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.258426 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:55.258650 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:55.259067 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 16:20:55.260313 (+ 1246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":158,"threads_started":1}
I20260929 16:20:55.262121 11044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } }
I20260929 16:20:55.262251 11044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:55.262750 11053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:55.262698 11042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128bfa733ff744bbbe6e105ec0ad0a62. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } }
I20260929 16:20:55.262820 11042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:55.265951 11053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:55.271885 11053 catalog_manager.cc:1347] Generated new cluster ID: 2ae92c7d808b46869257d3a8ee9603f6
I20260929 16:20:55.271973 11053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:55.279757 11053 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:20:55.281035 11053 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:55.287573 11053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Generated new TSK 0
I20260929 16:20:55.288379 11053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
I20260929 16:20:55.317440 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:55.431305 11063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:55.431630 11063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:55.431703 11063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:55.435904 11063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:55.436004 11063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:55.436102 11063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:55.441514 11063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:55.442726 11063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:55.443886 11063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:55.453912 11068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:55.454108 11063 server_base.cc:1034] running on GCE node
W20260929 16:20:55.453917 11069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.453917 11071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:55.454761 11063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:55.455438 11063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:55.456655 11063 hybrid_clock.cc:648] HybridClock initialized: now 1790698855456630 us; error 35 us; skew 500 ppm
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:55.459997 11063 init.cc:377] Logged in from keytab as oryx/127.4.200.65@KRBTEST.COM (short username oryx)
I20260929 16:20:55.460965 11063 webserver.cc:460] Webserver started at http://127.4.200.65:35725/ using document root <none> and password file <none>
I20260929 16:20:55.461617 11063 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:55.461709 11063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:55.461966 11063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:55.464102 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "dc685fd96de045cf8557c575cf51477a"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "48cd45be81dfb96f1f6a562c1fbd5a29"
server_key_iv: "c594294380c803e15012e4721f1fce6d"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.464672 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "dc685fd96de045cf8557c575cf51477a"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "48cd45be81dfb96f1f6a562c1fbd5a29"
server_key_iv: "c594294380c803e15012e4721f1fce6d"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.468461 11063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 16:20:55.471053 11078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.472295 11063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:55.472429 11063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "dc685fd96de045cf8557c575cf51477a"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "48cd45be81dfb96f1f6a562c1fbd5a29"
server_key_iv: "c594294380c803e15012e4721f1fce6d"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.472563 11063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:55.490603 11063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:55.493876 11063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:55.494102 11063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:55.494735 11063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:55.495678 11063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:55.495752 11063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.495820 11063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:55.495868 11063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.504928 11063 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44163
I20260929 16:20:55.504948 11191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44163 every 8 connection(s)
I20260929 16:20:55.505913 11063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:55.514654 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11063
I20260929 16:20:55.514799 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:20:55.515196 4897 external_mini_cluster.cc:1428] Setting key 62e76f94abf5934535407c0635977003
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:55.520458 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.507678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:58023 (local address 127.4.200.126:33163)
0929 16:20:55.507819 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:55.507823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:55.508902 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 16:20:55.509744 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:55.509746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:55.509793 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:55.509866 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:55.511184 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.511747 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.512872 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.513131 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.513218 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.516332 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:55.516336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:55.516339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:55.516364 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:55.518078 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.518762 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.518767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.518769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.518863 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.519247 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.519250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.519252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.519450 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:55.519568 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:55.520115 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 16:20:55.520251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:20:55.521340 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.508048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:58023)
0929 16:20:55.508759 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:55.508799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:55.509558 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:55.510036 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:55.510043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:55.510415 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:55.511027 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.511045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.511877 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.511881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.512756 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.512764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.513296 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.513300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.513470 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.514163 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:55.514181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:55.516155 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:55.518223 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.518230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.518247 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.518645 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.518975 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.518979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.518982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.519137 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.519569 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:55.519583 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:55.519854 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 16:20:55.520059 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:20:55.520426 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":127,"threads_started":1}
I20260929 16:20:55.522778 11192 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:55.523053 11192 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:55.523699 11192 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:20:55.525475 11001 ts_manager.cc:194] Registered new tserver with Master: dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163)
I20260929 16:20:55.526693 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:58023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
I20260929 16:20:55.546109 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:55.664382 11197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:55.664655 11197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:55.664700 11197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:55.668861 11197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:55.668926 11197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:55.669019 11197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:55.674377 11197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:55.675518 11197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:55.676589 11197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:55.686203 11205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.686209 11202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.686219 11203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:55.686326 11197 server_base.cc:1034] running on GCE node
I20260929 16:20:55.686975 11197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:55.687645 11197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:55.688812 11197 hybrid_clock.cc:648] HybridClock initialized: now 1790698855688785 us; error 38 us; skew 500 ppm
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:55.691639 11197 init.cc:377] Logged in from keytab as oryx/127.4.200.66@KRBTEST.COM (short username oryx)
I20260929 16:20:55.692550 11197 webserver.cc:460] Webserver started at http://127.4.200.66:43631/ using document root <none> and password file <none>
I20260929 16:20:55.693140 11197 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:55.693229 11197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:55.693461 11197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:55.695580 11197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "5218701608d04deca1175445803d1a10"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "52d78e967d3869aee95df7bc2c066e5c"
server_key_iv: "69bf6ec15c3638d6c8c163d822411e3c"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.696159 11197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "5218701608d04deca1175445803d1a10"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "52d78e967d3869aee95df7bc2c066e5c"
server_key_iv: "69bf6ec15c3638d6c8c163d822411e3c"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.700064 11197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:20:55.702811 11212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.704334 11197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:55.704484 11197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "5218701608d04deca1175445803d1a10"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "52d78e967d3869aee95df7bc2c066e5c"
server_key_iv: "69bf6ec15c3638d6c8c163d822411e3c"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.704619 11197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:55.718813 11197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:55.722038 11197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:55.722267 11197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:55.722972 11197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:55.723830 11197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:55.723902 11197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.723973 11197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:55.724017 11197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.733114 11197 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:43115
I20260929 16:20:55.733122 11325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:43115 every 8 connection(s)
I20260929 16:20:55.734045 11197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:55.743461 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11197
I20260929 16:20:55.743649 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:55.743992 4897 external_mini_cluster.cc:1428] Setting key 78fda4bc57124384c377dd96062c4476
I20260929 16:20:55.748433 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.735935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:41053 (local address 127.4.200.126:33163)
0929 16:20:55.736086 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:55.736090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:55.736916 (+ 826us) server_negotiation.cc:407] Connection header received
0929 16:20:55.738157 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:55.738160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:55.738208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:55.738285 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:55.739655 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.740199 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.741357 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.741613 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.741706 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.744557 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:55.744563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:55.744569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:55.744602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:55.746069 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.746743 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.746746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.746747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.746819 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.747238 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.747243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.747245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.747500 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:55.747607 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:55.748091 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 16:20:55.748237 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:55.749305 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.736330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:41053)
0929 16:20:55.736766 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:55.736807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:55.737939 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:55.738513 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:55.738521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:55.738884 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:55.739368 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.739380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.740337 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.740340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.741236 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.741246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.741757 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.741760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.741897 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.742565 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:55.742581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:55.744372 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:55.746191 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.746200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.746220 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.746651 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.746911 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.746917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.746922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.747114 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.747606 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:55.747611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:55.747843 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 16:20:55.747995 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:20:55.748249 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260929 16:20:55.750797 11326 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:55.751053 11326 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:55.751700 11326 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
I20260929 16:20:55.752993 11001 ts_manager.cc:194] Registered new tserver with Master: 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:55.753836 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:41053
WARNING: no policy specified for oryx/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
I20260929 16:20:55.774080 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:55.887256 11331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:55.887527 11331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:55.887573 11331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:55.891669 11331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:55.891726 11331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:55.891784 11331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:55.897105 11331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:55.898124 11331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:55.899192 11331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:55.909183 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.909303 11339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:55.909183 11336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:55.909433 11331 server_base.cc:1034] running on GCE node
I20260929 16:20:55.909830 11331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:55.910564 11331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:55.911759 11331 hybrid_clock.cc:648] HybridClock initialized: now 1790698855911747 us; error 34 us; skew 500 ppm
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:55.914652 11331 init.cc:377] Logged in from keytab as oryx/127.4.200.67@KRBTEST.COM (short username oryx)
I20260929 16:20:55.915536 11331 webserver.cc:460] Webserver started at http://127.4.200.67:42829/ using document root <none> and password file <none>
I20260929 16:20:55.916121 11331 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:55.916210 11331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:55.916450 11331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:55.918565 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "5f3a71ad6ca142f3862c9e493424ffd2"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "bd40609a5e295fae64a3377538b8d434"
server_key_iv: "0b07a6d865eddd30744e60ae8da04831"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.919135 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3a71ad6ca142f3862c9e493424ffd2"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "bd40609a5e295fae64a3377538b8d434"
server_key_iv: "0b07a6d865eddd30744e60ae8da04831"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.922755 11331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:20:55.925056 11346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.926271 11331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:55.926456 11331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "5f3a71ad6ca142f3862c9e493424ffd2"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "bd40609a5e295fae64a3377538b8d434"
server_key_iv: "0b07a6d865eddd30744e60ae8da04831"
server_key_version: "encryptionkey@0"
I20260929 16:20:55.926580 11331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:55.967898 11331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:55.970863 11331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:55.971069 11331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:55.971653 11331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:55.972471 11331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:55.972543 11331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.972615 11331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:55.972666 11331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:55.981870 11331 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:39947
I20260929 16:20:55.981906 11459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:39947 every 8 connection(s)
I20260929 16:20:55.982846 11331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:55.991628 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11331
I20260929 16:20:55.991788 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:20:55.992072 4897 external_mini_cluster.cc:1428] Setting key 976a4ab0740375844e891d5f1292fe1e
Sep 29 16:20:55 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:55.996699 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.984638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:33211 (local address 127.4.200.126:33163)
0929 16:20:55.984784 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:55.984787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:55.985722 (+ 935us) server_negotiation.cc:407] Connection header received
0929 16:20:55.987007 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:55.987010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:55.987057 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:55.987132 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:55.988250 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.988767 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.989836 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.990050 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.990133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.992884 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:55.992888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:55.992892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:55.992921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:55.994692 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.995216 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.995221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.995223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.995305 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:55.995653 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:55.995656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:55.995657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:55.995837 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:55.995943 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:55.996418 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 16:20:55.996547 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:20:55.997490 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:55.985159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:33211)
0929 16:20:55.985571 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:55.985612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:55.986809 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:55.987306 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:55.987313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:55.987656 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:55.988115 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.988128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.988867 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.988869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.989734 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:55.989743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:55.990190 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:55.990192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:55.990326 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:55.990957 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:55.990973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:55.992723 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:55.994809 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.994814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.994829 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.995109 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.995406 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:55.995409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:55.995411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:55.995549 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:55.995940 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:55.995959 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:55.996194 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 16:20:55.996380 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:20:55.996701 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260929 16:20:55.998668 11460 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:55.998915 11460 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:55.999400 11460 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
I20260929 16:20:56.000415 11001 ts_manager.cc:194] Registered new tserver with Master: 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:56.000965 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:33211
I20260929 16:20:56.006062 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.200.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.125.keytab.
Entry for principal oryx/127.4.200.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.125.keytab.
I20260929 16:20:56.031855 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:39541
--webserver_interface=127.4.200.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.125.keytab
--principal=oryx/127.4.200.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33163,127.4.200.125:39541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:20:56.145457 11465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:56.145737 11465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:56.145778 11465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:56.149942 11465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:56.150010 11465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:56.150030 11465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:56.150048 11465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:56.150063 11465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:56.155701 11465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33163,127.4.200.125:39541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.125.keytab
--principal=oryx/127.4.200.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.200.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
W20260929 16:20:56.155954 11465 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.200.126:33163,127.4.200.125:39541'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:20:56.156885 11465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:56.157963 11465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:56.166638 11473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.166677 11471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.166661 11470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.167045 11465 server_base.cc:1034] running on GCE node
I20260929 16:20:56.167636 11465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:56.168862 11465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:56.170076 11465 hybrid_clock.cc:648] HybridClock initialized: now 1790698856170056 us; error 39 us; skew 500 ppm
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:56.173476 11465 init.cc:377] Logged in from keytab as oryx/127.4.200.125@KRBTEST.COM (short username oryx)
I20260929 16:20:56.174381 11465 webserver.cc:460] Webserver started at http://127.4.200.125:34111/ using document root <none> and password file <none>
I20260929 16:20:56.175007 11465 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.175100 11465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.175313 11465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:56.177516 11465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/instance:
uuid: "97e744cba70f426b9666d9d85be8c1c2"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "7aa904c457ca84a1755220a9fe2b2e14"
server_key_iv: "ff972e7d55b91561ac88b9af6bfba92e"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.178102 11465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal/instance:
uuid: "97e744cba70f426b9666d9d85be8c1c2"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "7aa904c457ca84a1755220a9fe2b2e14"
server_key_iv: "ff972e7d55b91561ac88b9af6bfba92e"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.182004 11465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 16:20:56.184597 11480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.185879 11465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:20:56.186035 11465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "97e744cba70f426b9666d9d85be8c1c2"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "7aa904c457ca84a1755220a9fe2b2e14"
server_key_iv: "ff972e7d55b91561ac88b9af6bfba92e"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.186158 11465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.203186 11465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:56.206775 11465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:56.206988 11465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:56.213910 11465 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.125:39541
I20260929 16:20:56.214208 11532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.125:39541 every 8 connection(s)
I20260929 16:20:56.214962 11465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
I20260929 16:20:56.217823 11533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:56.218273 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11465
I20260929 16:20:56.222889 11533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.125@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:56.235540 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.225001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.4.200.126:33163)
0929 16:20:56.225136 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.225139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.225895 (+ 756us) server_negotiation.cc:407] Connection header received
0929 16:20:56.225988 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.225991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.226045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.226141 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.227000 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.227561 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.228705 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.229076 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.229177 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.231927 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.231930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.231932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.231957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.233601 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.234075 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.234079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.234081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.234183 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.234503 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.234507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.234509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.234803 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.234925 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.235267 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.235376 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:20:56.235592 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.224903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37350)
0929 16:20:56.225768 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.225806 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.225900 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.226191 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.226195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.226691 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.226871 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.226878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.227699 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.227705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.228567 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.228576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.229246 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.229250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.229382 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.230003 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.230015 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.231782 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.233705 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.233714 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.233720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.233941 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.234288 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.234290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.234292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.234365 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.234930 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.234935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.235031 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.235187 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.235375 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"spinlock_wait_cycles":768,"thread_start_us":122,"threads_started":1}
I20260929 16:20:56.235630 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.219845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.4.200.125:39541)
0929 16:20:56.220284 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.220303 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.220346 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:20:56.221143 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.221168 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.221605 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.221998 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.222883 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.224353 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.225647 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.226297 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.226474 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.229488 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.229496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.229516 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.229544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.231752 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.232219 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.232227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.232237 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.232386 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.232709 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.232713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.232716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.233276 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.233485 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.234213 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.234588 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260929 16:20:56.237764 11533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.125@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
I20260929 16:20:56.247167 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.238318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49352)
0929 16:20:56.238525 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.238539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.238614 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.238970 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.238973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.239152 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.239256 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.239262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.240304 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.240307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.241134 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.241143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.241631 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.241635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.241735 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.242221 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.242235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.243772 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.245667 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.245670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.245672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.245859 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.246132 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.246136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.246139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.246216 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.246645 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.246648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.246735 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.246840 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.246986 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 16:20:56.247205 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.238447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.4.200.125:39541)
0929 16:20:56.238767 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.238774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.238785 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:20:56.238814 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.238816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.238851 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.238927 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.239356 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.240201 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.241262 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.241507 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.241605 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.243915 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.243918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.243920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.243940 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.245545 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.245951 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.245958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.245959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.246036 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.246305 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.246307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.246309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.246544 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.246650 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.246883 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.247042 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":261,"thread_start_us":146,"threads_started":1}
I20260929 16:20:56.249890 11533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Bootstrap starting.
I20260929 16:20:56.252527 11533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:56.253234 11533 log.cc:824] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:56.255208 11533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: No bootstrap required, opened a new log
I20260929 16:20:56.258268 11533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } }
I20260929 16:20:56.258558 11533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:56.258599 11533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e744cba70f426b9666d9d85be8c1c2, State: Initialized, Role: FOLLOWER
I20260929 16:20:56.259048 11533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } }
I20260929 16:20:56.259539 11540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } } }
I20260929 16:20:56.259661 11540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:56.260277 11533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:20:56.265751 11551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:56.265854 11551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:56.267172 11465 master_runner.cc:428] Detected that this master 97e744cba70f426b9666d9d85be8c1c2 is joining an existing cluster
I20260929 16:20:56.267253 11465 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.200.125:39541
I20260929 16:20:56.278441 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.269413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.4.200.126:33163)
0929 16:20:56.269546 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.269551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.269808 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:20:56.269941 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.269943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.269995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.270084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.270794 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.271604 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.272790 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.273162 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.273278 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.274717 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.274722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.274725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.274752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.276549 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277132 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.277135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.277137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.277229 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277468 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.277471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.277473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.277732 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.277838 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.278120 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.278256 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:56.278448 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.269794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.4.200.125:39541)
0929 16:20:56.269897 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.269902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.270314 (+ 412us) server_negotiation.cc:407] Connection header received
0929 16:20:56.270434 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.270437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.270480 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.270557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.271088 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.271822 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.273070 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.273413 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.273521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.275069 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.275073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.275075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.275101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.276742 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277229 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.277232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.277234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.277306 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277601 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.277604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.277606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.277808 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.277890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.278158 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.278257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":8064}
I20260929 16:20:56.278440 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.269321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37356)
0929 16:20:56.269726 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.269743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.269849 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.270099 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.270102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.270398 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.270583 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.270590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.271721 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.271725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.272646 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.272656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.273291 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.273294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.273380 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.273881 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.273918 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.274558 (+ 640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.276689 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.276694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.276696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.277025 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.277305 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.277311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.277392 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.277831 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.277834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.277926 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.278040 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.278180 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"spinlock_wait_cycles":9344,"thread_start_us":106,"threads_started":1}
I20260929 16:20:56.278820 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.269733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49360)
0929 16:20:56.270212 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.270226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.270328 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.270600 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.270604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.270800 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.270953 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.270961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.271927 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.271932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.272955 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.272964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.273534 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.273538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.273647 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.273980 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.273996 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.274558 (+ 562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.276842 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.276845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.276847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.277128 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.277399 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.277402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.277404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.277497 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.277922 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.277930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.277991 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.278464 (+ 473us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.278605 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"spinlock_wait_cycles":1792,"thread_start_us":107,"threads_started":1}
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.126@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
I20260929 16:20:56.293579 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.283841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.4.200.125:39541)
0929 16:20:56.283984 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.283989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.284389 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:20:56.284557 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.284559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.284605 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.284673 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.285450 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.285951 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.287213 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.287420 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.287503 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.290114 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.290118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.290120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.290143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.291755 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.292252 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.292259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.292263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.292348 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.292631 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.292635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.292637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.292819 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.292926 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.293270 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.293406 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:56.293582 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.283802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49374)
0929 16:20:56.284241 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.284302 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.284442 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.284727 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.284734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.285201 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.285334 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.285345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.286053 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.286057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.287090 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.287101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.287531 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.287535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.287648 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.288172 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.288188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.289934 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.291880 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.291891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.291898 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.292142 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.292443 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.292447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.292449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.292537 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.292926 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.292935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.293059 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.293205 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.293411 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":119,"threads_started":1}
I20260929 16:20:56.294977 11001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.200.125:39541: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2"
member_type: NON_VOTER
last_known_addr {
host: "127.4.200.125"
port: 39541
}
attrs {
promote: true
}
}
dest_uuid: "128bfa733ff744bbbe6e105ec0ad0a62"
cas_config_opid_index: -1
I20260929 16:20:56.295670 11001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } }
I20260929 16:20:56.298120 11507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:56.297592 11561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } }
I20260929 16:20:56.299607 11507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Refusing update from remote peer 128bfa733ff744bbbe6e105ec0ad0a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:20:56.300128 11561 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.200.125:39541
I20260929 16:20:56.300652 11561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:56.300732 11465 master.cc:537] Master@127.4.200.125:39541 shutting down...
I20260929 16:20:56.301095 11564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } }
I20260929 16:20:56.301210 11564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
W20260929 16:20:56.304206 10987 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 16:20:56.305074 10987 consensus_peers.cc:487] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 -> Peer 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125:39541): Couldn't send request to peer 97e744cba70f426b9666d9d85be8c1c2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.4.200.124@KRBTEST.COM; defaulting to no policy
I20260929 16:20:56.306226 11465 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:56.306540 11465 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:56.306653 11465 tablet_replica.cc:330] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.124@KRBTEST.COM" created.
I20260929 16:20:56.323710 11465 master.cc:552] Master@127.4.200.125:39541 shutdown complete.
I20260929 16:20:56.323782 11465 master_runner.cc:305] Clearing existing system tablet
I20260929 16:20:56.324046 11465 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.324103 11465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.325440 11465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 16:20:56.326352 11569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.124.keytab.
Entry for principal oryx/127.4.200.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.124.keytab.
I20260929 16:20:56.326581 11465 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.326753 11465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:56.326805 11465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "97e744cba70f426b9666d9d85be8c1c2"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "7aa904c457ca84a1755220a9fe2b2e14"
server_key_iv: "ff972e7d55b91561ac88b9af6bfba92e"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.327494 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:37523
--webserver_interface=127.4.200.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.124.keytab
--principal=oryx/127.4.200.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:20:56.328648 11465 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:56.331710 11465 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:20:56.331812 11465 log.cc:1196] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:20:56.332106 11465 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Deleting consensus metadata
I20260929 16:20:56.332314 11465 master_runner.cc:315] Copying system tablet from 127.4.200.126:33163
I20260929 16:20:56.333760 11465 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.200.126:33163
I20260929 16:20:56.347090 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.334378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37358)
0929 16:20:56.338638 (+ 4260us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.338657 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.338809 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.339124 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.339128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.339347 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.339493 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.339500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.340606 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.340609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.341517 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.341527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.342121 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.342125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.342228 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.342836 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.342850 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.343288 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.345411 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.345414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.345417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.345673 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.345977 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.345980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.345983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.346072 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.346505 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.346508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.346613 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.346721 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.346873 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4167,"thread_start_us":100,"threads_started":1}
I20260929 16:20:56.347095 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.334571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.4.200.126:33163)
0929 16:20:56.334741 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.334748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.338794 (+ 4046us) server_negotiation.cc:407] Connection header received
0929 16:20:56.338904 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.338908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.338967 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.339070 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.339621 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.340477 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.341643 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.342001 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.342105 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.343445 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.343449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.343452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.343481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.345287 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.345778 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.345782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.345783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.345874 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.346170 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.346173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.346174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.346422 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.346520 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.346789 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.346938 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:56.348166 11021 tablet_copy_service.cc:140] P 128bfa733ff744bbbe6e105ec0ad0a62: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 97e744cba70f426b9666d9d85be8c1c2 ({username='oryx', principal='oryx/127.4.200.125@KRBTEST.COM'} at 127.0.0.1:37358)
I20260929 16:20:56.348374 11021 tablet_copy_service.cc:161] P 128bfa733ff744bbbe6e105ec0ad0a62: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 97e744cba70f426b9666d9d85be8c1c2 at {username='oryx', principal='oryx/127.4.200.125@KRBTEST.COM'} at 127.0.0.1:37358: session id = 97e744cba70f426b9666d9d85be8c1c2-00000000000000000000000000000000
I20260929 16:20:56.350481 11021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:20:56.352521 11465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:20:56.355072 11465 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: tablet copy: Starting download of 0 data blocks...
I20260929 16:20:56.355285 11465 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: tablet copy: Starting download of 1 WAL segments...
I20260929 16:20:56.357317 11465 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:20:56.359333 11021 tablet_copy_service.cc:342] P 128bfa733ff744bbbe6e105ec0ad0a62: Request end of tablet copy session 97e744cba70f426b9666d9d85be8c1c2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.200.125@KRBTEST.COM'} at 127.0.0.1:37358
I20260929 16:20:56.359486 11021 tablet_copy_service.cc:433] P 128bfa733ff744bbbe6e105ec0ad0a62: ending tablet copy session 97e744cba70f426b9666d9d85be8c1c2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 97e744cba70f426b9666d9d85be8c1c2
I20260929 16:20:56.367220 11465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:56.369241 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.369309 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.371351 11465 server_base.cc:1034] running on GCE node
W20260929 16:20:56.371634 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.371884 11465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:56.372205 11465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:56.373351 11465 hybrid_clock.cc:648] HybridClock initialized: now 1790698856373327 us; error 42 us; skew 500 ppm
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:56.375129 11465 init.cc:377] Logged in from keytab as oryx/127.4.200.125@KRBTEST.COM (short username oryx)
I20260929 16:20:56.375686 11465 webserver.cc:460] Webserver started at http://127.4.200.125:45293/ using document root <none> and password file <none>
I20260929 16:20:56.375923 11465 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.376022 11465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.377471 11465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:56.378551 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.378829 11465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:56.378897 11465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "97e744cba70f426b9666d9d85be8c1c2"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "7aa904c457ca84a1755220a9fe2b2e14"
server_key_iv: "ff972e7d55b91561ac88b9af6bfba92e"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.378973 11465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.398949 11465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:56.400866 11465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:56.405961 11465 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.125:39541
I20260929 16:20:56.406517 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.125:39541 every 8 connection(s)
I20260929 16:20:56.406643 11465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
I20260929 16:20:56.408166 11640 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:20:56.408964 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Bootstrap starting.
I20260929 16:20:56.417074 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:20:56.417403 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: Bootstrap complete.
I20260929 16:20:56.417902 11640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } }
I20260929 16:20:56.418045 11640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97e744cba70f426b9666d9d85be8c1c2, State: Initialized, Role: LEARNER
I20260929 16:20:56.418159 11640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } }
I20260929 16:20:56.418529 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } }
I20260929 16:20:56.418845 11640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:56.418861 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:20:56.422349 11654 catalog_manager.cc:1259] Loaded cluster ID: 2ae92c7d808b46869257d3a8ee9603f6
I20260929 16:20:56.422458 11654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: loading cluster ID for follower catalog manager: success
I20260929 16:20:56.424063 11654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: acquiring CA information for follower catalog manager: success
I20260929 16:20:56.425256 11654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:20:56.478624 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:56.479063 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:56.479140 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:56.483407 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:20:56.483472 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:56.483492 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:56.483510 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:20:56.483525 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:20:56.489167 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.124.keytab
--principal=oryx/127.4.200.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.200.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:56.490274 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:56.491336 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:56.500214 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.500193 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.500218 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.500464 11570 server_base.cc:1034] running on GCE node
I20260929 16:20:56.503019 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:56.504726 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:56.508517 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790698856508485 us; error 47 us; skew 500 ppm
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:56.511572 11570 init.cc:377] Logged in from keytab as oryx/127.4.200.124@KRBTEST.COM (short username oryx)
I20260929 16:20:56.512494 11570 webserver.cc:460] Webserver started at http://127.4.200.124:41267/ using document root <none> and password file <none>
I20260929 16:20:56.513072 11570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.513131 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.513342 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:20:56.515555 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/instance:
uuid: "6ec00ccb6715431894a52d2d24ba4b89"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "9ad4214adaa78febedf15c3324cb1e01"
server_key_iv: "0d37daa5d85598f92e09a755c08d7088"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.516127 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal/instance:
uuid: "6ec00ccb6715431894a52d2d24ba4b89"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "9ad4214adaa78febedf15c3324cb1e01"
server_key_iv: "0d37daa5d85598f92e09a755c08d7088"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.520159 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:20:56.522821 11669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.524194 11570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:56.524353 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "6ec00ccb6715431894a52d2d24ba4b89"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "9ad4214adaa78febedf15c3324cb1e01"
server_key_iv: "0d37daa5d85598f92e09a755c08d7088"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.524472 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.529237 11192 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
I20260929 16:20:56.566490 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:56.570274 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:56.570524 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:56.577647 11570 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.124:37523
I20260929 16:20:56.577966 11721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.124:37523 every 8 connection(s)
I20260929 16:20:56.578748 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
I20260929 16:20:56.581544 11722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:56.585491 11722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:20:56.586122 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11570
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.124@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
I20260929 16:20:56.598920 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.587607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.4.200.126:33163)
0929 16:20:56.587757 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.587762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.588275 (+ 513us) server_negotiation.cc:407] Connection header received
0929 16:20:56.589205 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.589208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.589266 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.589350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.590187 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.590816 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.591832 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.592135 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.592247 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.594812 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.594815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.594818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.594846 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.596683 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.597311 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.597318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.597321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.597414 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.597761 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.597766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.597767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.597964 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.598069 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.598611 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.598745 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:20:56.599874 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.587660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.4.200.124:37523)
0929 16:20:56.588127 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.588144 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.588184 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:20:56.588924 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.588943 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.589162 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.589352 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.590202 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.591126 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.591953 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.592319 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.592528 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.595326 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.595335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.595350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.595375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.597274 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.597718 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.597735 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.597736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.597808 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.598123 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.598127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.598129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.598376 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.598661 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.598857 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.599141 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":896,"thread_start_us":190,"threads_started":1}
I20260929 16:20:56.599875 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.587646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37362)
0929 16:20:56.588146 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.588183 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.588950 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.589409 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.589415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.589784 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.590034 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.590047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.590946 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.590954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.591706 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.591715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.592285 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.592289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.592438 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.593001 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.593017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.594612 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.596810 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.596817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.596825 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.597207 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.597516 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.597521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.597523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.597661 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.598076 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.598085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.598344 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.598572 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.598886 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":2,"thread_start_us":128,"threads_started":1}
I20260929 16:20:56.601704 11722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.124@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
I20260929 16:20:56.610013 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.602201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49390)
0929 16:20:56.602577 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.602589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.602651 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.602871 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.602873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.602999 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.603084 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.603089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.604016 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.604018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.604623 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.604632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.605078 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.605081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.605159 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.605765 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.605776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.607025 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.608715 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.608717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.608719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.608993 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.609250 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.609252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.609254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.609324 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.609641 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.609643 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.609713 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.609805 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.609901 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260929 16:20:56.610066 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.602326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.4.200.125:39541)
0929 16:20:56.602624 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.602628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.602641 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:20:56.602731 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.602733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.602774 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.602846 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.603220 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.603913 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.604750 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.604986 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.605069 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.607152 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.607155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.607157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.607181 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.608616 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.609080 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.609083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.609084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.609158 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.609396 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.609398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.609399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.609562 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.609638 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.609857 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.609950 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
I20260929 16:20:56.610914 11722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.124@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
I20260929 16:20:56.618806 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.611393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.4.200.124:37523)
0929 16:20:56.611501 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.611506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.611560 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:20:56.611682 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.611685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.611721 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.611799 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.612128 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.612607 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.613318 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.613543 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.613644 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.615488 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.615491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.615493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.615512 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.617258 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.617640 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.617647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.617648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.617723 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.617986 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.617989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.617990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.618155 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.618252 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.618534 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.618677 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":43}
I20260929 16:20:56.618806 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.611383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55196)
0929 16:20:56.611501 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.611516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.611602 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.611807 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.611810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.611964 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.612021 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.612026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.612716 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.612719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.613211 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.613217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.613649 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.613652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.613723 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.614151 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.614164 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.615354 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.617356 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.617362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.617366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.617544 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.617814 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.617817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.617818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.617898 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.618251 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.618254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.618330 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.618488 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.618625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:20:56.622713 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Bootstrap starting.
I20260929 16:20:56.624892 11722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:56.625559 11722 log.cc:824] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:56.627420 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: No bootstrap required, opened a new log
I20260929 16:20:56.630518 11722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } }
I20260929 16:20:56.630709 11722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:56.630748 11722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec00ccb6715431894a52d2d24ba4b89, State: Initialized, Role: FOLLOWER
I20260929 16:20:56.631222 11722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } }
I20260929 16:20:56.631685 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } } }
I20260929 16:20:56.631819 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:56.632390 11722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:20:56.637077 11739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:20:56.637176 11739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:20:56.640579 11570 master_runner.cc:428] Detected that this master 6ec00ccb6715431894a52d2d24ba4b89 is joining an existing cluster
I20260929 16:20:56.640650 11570 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.200.124:37523
I20260929 16:20:56.652249 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.644088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.4.200.125:39541)
0929 16:20:56.644232 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.644236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.644386 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:20:56.644522 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.644525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.644573 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.644622 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.645194 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.645962 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647266 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.647573 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647673 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.648935 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.648939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.648942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.648969 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.650557 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.650940 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.650943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.650946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.651033 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.651297 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.651301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.651305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.651513 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.651616 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.651928 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.652089 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:20:56.652878 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.643395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49394)
0929 16:20:56.644300 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.644310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.644413 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.644752 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.644755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.644955 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.645054 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.645059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.646093 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.646096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.647143 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.647151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.647851 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.647999 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.648466 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.648477 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.648796 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.650656 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.650659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.650661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.650845 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.651137 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.651140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.651141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.651214 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.651619 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.651622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.651738 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.652528 (+ 790us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.652691 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":87,"thread_start_us":90,"threads_started":1}
I20260929 16:20:56.653481 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11063
I20260929 16:20:56.659938 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:44163
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:33163
--webserver_port=35725
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:20:56.671169 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.643523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.4.200.124:37523)
0929 16:20:56.643641 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.643646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.643662 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:20:56.643805 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.643808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.643857 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.643938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.644920 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.645669 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647300 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.647609 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647692 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.667728 (+ 20036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.667731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.667734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.667755 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.669423 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.669893 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.669896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.669898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.669962 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.670204 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.670207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.670209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.670491 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.670575 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.670841 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.670961 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:20:56.671701 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.642997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55208)
0929 16:20:56.643603 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.643614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.643700 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.644502 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.644506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.644704 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.644805 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.644812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.645789 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.645792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.647181 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.647191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.647838 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.647959 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.667051 (+ 19092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.667071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.667565 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.669567 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.669569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.669572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.669799 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.670053 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.670055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.670057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.670125 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.670610 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.670613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.670695 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.671387 (+ 692us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.671540 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":54,"thread_start_us":88,"threads_started":1}
I20260929 16:20:56.672472 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.642964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37374)
0929 16:20:56.643274 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.643290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.643386 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.643975 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.643979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.644241 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.644365 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.644373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.645390 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.645393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.646349 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.646361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.647096 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.647099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.647190 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.647773 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.647792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.667248 (+ 19456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.670209 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.670214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.670219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.670554 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.670958 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.670963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.670967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.671070 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.671962 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.671965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.672062 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.672161 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.672297 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":41,"thread_start_us":122,"threads_started":1}
I20260929 16:20:56.673270 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.643396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.4.200.126:33163)
0929 16:20:56.643730 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.643733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.643742 (+ 9us) server_negotiation.cc:407] Connection header received
0929 16:20:56.643805 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.643807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.643857 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.643938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.644477 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.645277 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.646510 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.646865 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.646939 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.667519 (+ 20580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.667527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.667533 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.667576 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.670044 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.670712 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.670717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.670721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.670831 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.671186 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.671189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.671191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.671411 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.671499 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.672989 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.673118 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260929 16:20:56.674661 11728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:56.674815 11728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } }
I20260929 16:20:56.676363 11728 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163), 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125:39541)
I20260929 16:20:56.677074 11011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec00ccb6715431894a52d2d24ba4b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" is_pre_election: true
I20260929 16:20:56.677577 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec00ccb6715431894a52d2d24ba4b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e744cba70f426b9666d9d85be8c1c2" is_pre_election: true
I20260929 16:20:56.677757 11614 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Handling vote request from an unknown peer 6ec00ccb6715431894a52d2d24ba4b89
I20260929 16:20:56.677882 11614 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 6ec00ccb6715431894a52d2d24ba4b89 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 16:20:56.678236 11672 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ec00ccb6715431894a52d2d24ba4b89; no voters: 128bfa733ff744bbbe6e105ec0ad0a62, 97e744cba70f426b9666d9d85be8c1c2
I20260929 16:20:56.679023 11728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:56.680334 11728 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.126@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
I20260929 16:20:56.689334 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.678266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55212)
0929 16:20:56.678685 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.678703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.678796 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.680528 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.680531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.680756 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.680865 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.680872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.681924 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.681928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.682874 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.682885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.683443 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.683446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.683543 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.684099 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.684115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.685715 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.687722 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.687726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.687728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.687977 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.688270 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.688273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.688275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.688369 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.688805 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.688808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.688914 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.689021 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.689168 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260929 16:20:56.689352 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.679035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.4.200.124:37523)
0929 16:20:56.679140 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.679145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.679157 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:56.679238 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.679241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.679292 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.679369 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.680986 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.681797 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.682995 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.683320 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.683425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.685864 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.685868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.685872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.685898 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.687600 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.688081 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.688085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.688087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.688171 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.688473 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.688477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.688479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.688702 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.688799 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.689087 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.689197 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:20:56.690564 11001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.200.124:37523: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89"
member_type: NON_VOTER
last_known_addr {
host: "127.4.200.124"
port: 37523
}
attrs {
promote: true
}
}
dest_uuid: "128bfa733ff744bbbe6e105ec0ad0a62"
cas_config_opid_index: 5
I20260929 16:20:56.691161 11001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } }
I20260929 16:20:56.692212 11564 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.693749 11564 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.200.124:37523
I20260929 16:20:56.693902 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.694007 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:56.695021 11570 master.cc:537] Master@127.4.200.124:37523 shutting down...
W20260929 16:20:56.697877 10986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 -> Peer 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124:37523): Couldn't send request to peer 6ec00ccb6715431894a52d2d24ba4b89. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:20:56.698527 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.692618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.4.200.125:39541)
0929 16:20:56.693989 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.693994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.694008 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:56.694057 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.694059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.694118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.694235 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:56.694560 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.695344 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.697036 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.697701 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.697781 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.697930 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.698285 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.698365 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288}
I20260929 16:20:56.698899 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.692383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49400)
0929 16:20:56.692832 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.692845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.692911 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.694273 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.694276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.694295 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:56.694382 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.694388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.695494 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.695498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.696892 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.696903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.698029 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.698032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.698128 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.698172 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.698712 (+ 540us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.698790 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379}
I20260929 16:20:56.699381 11614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Refusing update from remote peer 128bfa733ff744bbbe6e105ec0ad0a62: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:20:56.699972 11561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:20:56.701714 11570 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:56.701987 11570 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:56.702076 11570 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: stopping tablet replica
I20260929 16:20:56.701887 11614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.704294 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128bfa733ff744bbbe6e105ec0ad0a62. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.704424 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:20:56.704514 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: NON_VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: true } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.704573 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:20:56.705428 11564 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: attempting to promote NON_VOTER 97e744cba70f426b9666d9d85be8c1c2 to VOTER
I20260929 16:20:56.705920 11564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } }
I20260929 16:20:56.707888 11614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 LEARNER]: Refusing update from remote peer 128bfa733ff744bbbe6e105ec0ad0a62: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 16:20:56.708319 11561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 16:20:56.709766 11570 master.cc:552] Master@127.4.200.124:37523 shutdown complete.
I20260929 16:20:56.709859 11570 master_runner.cc:305] Clearing existing system tablet
I20260929 16:20:56.710131 11570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.710222 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.710634 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.710285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55226)
0929 16:20:56.710458 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.710547 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 16:20:56.710767 10986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 -> Peer 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124:37523): Couldn't send request to peer 6ec00ccb6715431894a52d2d24ba4b89. Status: Network error: Client connection negotiation failed: client connection to 127.4.200.124:37523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:20:56.711625 11564 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.712039 11614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.713438 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.713577 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:56.713541 11562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.713665 11562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:56.714025 11562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.714123 11562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:56.714489 11570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260929 16:20:56.715648 11753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.715885 11570 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.716184 11570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:20:56.716265 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "6ec00ccb6715431894a52d2d24ba4b89"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "9ad4214adaa78febedf15c3324cb1e01"
server_key_iv: "0d37daa5d85598f92e09a755c08d7088"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.718372 11570 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:56.721081 11570 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:20:56.721299 11570 log.cc:1196] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:20:56.721727 11570 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Deleting consensus metadata
I20260929 16:20:56.721992 11570 master_runner.cc:315] Copying system tablet from 127.4.200.126:33163
I20260929 16:20:56.723353 11570 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: tablet copy: Beginning tablet copy session from remote peer at address 127.4.200.126:33163
I20260929 16:20:56.732856 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.724106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.4.200.126:33163)
0929 16:20:56.724258 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.724264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.724422 (+ 158us) server_negotiation.cc:407] Connection header received
0929 16:20:56.724541 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.724544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.724600 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.724708 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.725218 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.726022 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.727247 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.727575 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.727695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.729024 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.729028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.729031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.729060 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.730909 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.731431 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.731436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.731439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.731551 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.731899 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.731902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.731903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.732134 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.732228 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.732566 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.732704 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:20:56.732869 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.723992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37382)
0929 16:20:56.724316 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.724332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.724451 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.724743 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.724746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.724976 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.725088 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.725095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.726157 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.726161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.727109 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.727120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.727704 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.727708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.727798 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.728389 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.728404 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.728861 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.731042 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.731045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.731047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.731309 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.731669 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.731674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.731679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.731793 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.732280 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.732283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.732382 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.732490 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.732651 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260929 16:20:56.733743 11021 tablet_copy_service.cc:140] P 128bfa733ff744bbbe6e105ec0ad0a62: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6ec00ccb6715431894a52d2d24ba4b89 ({username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:37382)
I20260929 16:20:56.733851 11021 tablet_copy_service.cc:161] P 128bfa733ff744bbbe6e105ec0ad0a62: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6ec00ccb6715431894a52d2d24ba4b89 at {username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:37382: session id = 6ec00ccb6715431894a52d2d24ba4b89-00000000000000000000000000000000
I20260929 16:20:56.735013 11021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:20:56.736332 11570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:20:56.738912 11570 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: tablet copy: Starting download of 0 data blocks...
I20260929 16:20:56.739182 11570 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: tablet copy: Starting download of 1 WAL segments...
I20260929 16:20:56.740866 11570 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:20:56.742761 11021 tablet_copy_service.cc:342] P 128bfa733ff744bbbe6e105ec0ad0a62: Request end of tablet copy session 6ec00ccb6715431894a52d2d24ba4b89-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:37382
I20260929 16:20:56.742851 11021 tablet_copy_service.cc:433] P 128bfa733ff744bbbe6e105ec0ad0a62: ending tablet copy session 6ec00ccb6715431894a52d2d24ba4b89-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6ec00ccb6715431894a52d2d24ba4b89
I20260929 16:20:56.750192 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:56.752063 11760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.752216 11761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.752557 11763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.752555 11570 server_base.cc:1034] running on GCE node
I20260929 16:20:56.753014 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:56.753367 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:56.754560 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790698856754541 us; error 37 us; skew 500 ppm
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:56.756183 11326 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
I20260929 16:20:56.756428 11570 init.cc:377] Logged in from keytab as oryx/127.4.200.124@KRBTEST.COM (short username oryx)
I20260929 16:20:56.756930 11570 webserver.cc:460] Webserver started at http://127.4.200.124:43927/ using document root <none> and password file <none>
I20260929 16:20:56.757179 11570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.757277 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.758697 11570 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:56.759738 11770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.759997 11570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:20:56.760102 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "6ec00ccb6715431894a52d2d24ba4b89"
format_stamp: "Formatted at 2026-09-29 16:20:56 on dist-test-slave-s9ct"
server_key: "9ad4214adaa78febedf15c3324cb1e01"
server_key_iv: "0d37daa5d85598f92e09a755c08d7088"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.760244 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.775867 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:56.778245 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:56.783624 11570 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.124:37523
I20260929 16:20:56.783659 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.124:37523 every 8 connection(s)
I20260929 16:20:56.784252 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
I20260929 16:20:56.786224 11823 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:20:56.787376 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Bootstrap starting.
I20260929 16:20:56.799104 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:20:56.799453 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Bootstrap complete.
I20260929 16:20:56.800092 11823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } }
I20260929 16:20:56.800246 11823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ec00ccb6715431894a52d2d24ba4b89, State: Initialized, Role: LEARNER
I20260929 16:20:56.800356 11823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } }
I20260929 16:20:56.800735 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:56.800871 11823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:20:56.800947 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:20:56.804397 11837 catalog_manager.cc:1259] Loaded cluster ID: 2ae92c7d808b46869257d3a8ee9603f6
I20260929 16:20:56.804584 11837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: loading cluster ID for follower catalog manager: success
I20260929 16:20:56.806756 11837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: acquiring CA information for follower catalog manager: success
I20260929 16:20:56.808076 11837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:20:56.821049 11748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:56.821350 11748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:56.821439 11748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:56.825773 11748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:56.825871 11748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:56.825973 11748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:20:56.831776 11748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=35725
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:56.832911 11748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:56.833983 11748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:56.844676 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:56.844791 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.844717 11748 server_base.cc:1034] running on GCE node
W20260929 16:20:56.844723 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:56.845428 11748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:56.846201 11748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:56.847402 11748 hybrid_clock.cc:648] HybridClock initialized: now 1790698856847388 us; error 35 us; skew 500 ppm
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:56.850586 11748 init.cc:377] Logged in from keytab as oryx/127.4.200.65@KRBTEST.COM (short username oryx)
I20260929 16:20:56.851531 11748 webserver.cc:460] Webserver started at http://127.4.200.65:35725/ using document root <none> and password file <none>
I20260929 16:20:56.852133 11748 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:56.852238 11748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:56.856422 11748 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:20:56.858844 11852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.860266 11748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:20:56.860415 11748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "dc685fd96de045cf8557c575cf51477a"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "48cd45be81dfb96f1f6a562c1fbd5a29"
server_key_iv: "c594294380c803e15012e4721f1fce6d"
server_key_version: "encryptionkey@0"
I20260929 16:20:56.861083 11748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:56.880703 11748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:56.884452 11748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:56.884613 11748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:56.885174 11748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:56.886144 11748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:56.886195 11748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.886235 11748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:56.886251 11748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:56.896467 11748 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44163
I20260929 16:20:56.896488 11965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44163 every 8 connection(s)
I20260929 16:20:56.897624 11748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:20:56.897909 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11748
I20260929 16:20:56.898120 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11197
I20260929 16:20:56.905396 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:43115
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:33163
--webserver_port=43631
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
Sep 29 16:20:56 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:56.917315 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.899636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:34581 (local address 127.4.200.124:37523)
0929 16:20:56.900301 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.900308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.901272 (+ 964us) server_negotiation.cc:407] Connection header received
0929 16:20:56.902760 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.902764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.902825 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.902879 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.905205 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906275 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908105 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908458 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908537 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.912903 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.912907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.912910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.912941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.914687 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915275 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.915278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.915280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.915362 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915799 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.915805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.915810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.916087 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.916182 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.916955 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.917112 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":83,"threads_started":1}
I20260929 16:20:56.917341 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.902603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:43509 (local address 127.4.200.125:39541)
0929 16:20:56.902876 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.902883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.903556 (+ 673us) server_negotiation.cc:407] Connection header received
0929 16:20:56.903739 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.903745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.903806 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.903861 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.905212 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906015 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908097 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908484 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.912659 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.912663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.912668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.912703 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.914999 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915581 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.915587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.915591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.915686 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.916037 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.916043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.916047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.916310 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.916407 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.916942 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.917087 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260929 16:20:56.918785 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.902441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:47585 (local address 127.4.200.126:33163)
0929 16:20:56.902763 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:56.902770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:56.903237 (+ 467us) server_negotiation.cc:407] Connection header received
0929 16:20:56.903341 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:56.903344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:56.903392 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:56.903477 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:56.905205 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906024 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908105 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908458 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908537 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.913338 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:56.913342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:56.913348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:56.913382 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:56.916932 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.917423 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.917426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.917428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.917518 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:56.917821 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:56.917824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:56.917826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:56.918032 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:56.918125 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:56.918496 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:20:56.918629 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260929 16:20:56.919150 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.900130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.4.200.65:34581)
0929 16:20:56.901119 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.901172 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.902384 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.903314 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.903324 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.904105 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.905028 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.905048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906636 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.906639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.907967 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.907977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908735 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.909024 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.909983 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.909995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.912757 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.914816 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.914822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.914840 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.915164 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.915474 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.915485 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.915680 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.916261 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.916270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.916620 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.917545 (+ 925us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.917927 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":458,"spinlock_wait_cycles":11904,"thread_start_us":614,"threads_started":1}
I20260929 16:20:56.919341 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.901772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:47585)
0929 16:20:56.903117 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.903144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.903251 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.903607 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.903610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.904105 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.905028 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.905048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906188 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.906191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.907967 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.907977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908735 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.909065 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.909916 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.909943 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.913146 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.917064 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.917069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.917073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.917304 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.917640 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.917642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.917644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.917723 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.918179 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.918182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.918261 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.918984 (+ 723us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.919120 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":484,"spinlock_wait_cycles":34176,"thread_start_us":94,"threads_started":1}
I20260929 16:20:56.919894 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:56.902147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.4.200.65:43509)
0929 16:20:56.903480 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:56.903494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:56.903633 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:56.904012 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:56.904017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:56.904270 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:56.905048 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.905058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.906253 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.906258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.907953 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:56.907967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:56.908757 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:56.908763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:56.909061 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:56.910200 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:56.910221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:56.912401 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:56.915163 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.915174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.915451 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.915815 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:56.915820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:56.915824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:56.915921 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:56.916476 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:56.916482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:56.916624 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 16:20:56.918065 (+ 1441us) client_negotiation.cc:240] Negotiation successful
0929 16:20:56.918239 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":253,"spinlock_wait_cycles":30976,"thread_start_us":69,"threads_started":1}
I20260929 16:20:56.920717 11966 heartbeater.cc:347] Connected to a master server at 127.4.200.124:37523
I20260929 16:20:56.920861 11967 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:56.920970 11966 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:56.920969 11967 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:56.921381 11968 heartbeater.cc:347] Connected to a master server at 127.4.200.125:39541
I20260929 16:20:56.921452 11968 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:56.921638 11966 heartbeater.cc:511] Master 127.4.200.124:37523 requested a full tablet report, sending...
I20260929 16:20:56.921638 11968 heartbeater.cc:511] Master 127.4.200.125:39541 requested a full tablet report, sending...
I20260929 16:20:56.921638 11967 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
I20260929 16:20:56.923210 11001 ts_manager.cc:194] Re-registered known tserver with Master: dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163)
I20260929 16:20:56.923394 11604 ts_manager.cc:194] Registered new tserver with Master: dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163)
I20260929 16:20:56.923439 11787 ts_manager.cc:194] Registered new tserver with Master: dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163)
I20260929 16:20:56.923986 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:47585
I20260929 16:20:57.003080 11460 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
W20260929 16:20:57.030215 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:57.030606 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:57.030700 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:57.034699 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:57.034792 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:57.034890 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:20:57.040205 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:43115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=43631
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:57.041297 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:57.042335 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:57.052264 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:57.052266 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:57.052264 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:57.052721 11974 server_base.cc:1034] running on GCE node
I20260929 16:20:57.053201 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:57.053794 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:57.054960 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790698857054938 us; error 34 us; skew 500 ppm
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:57.057664 11974 init.cc:377] Logged in from keytab as oryx/127.4.200.66@KRBTEST.COM (short username oryx)
I20260929 16:20:57.058725 11974 webserver.cc:460] Webserver started at http://127.4.200.66:43631/ using document root <none> and password file <none>
I20260929 16:20:57.059322 11974 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:57.059435 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:57.063951 11974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:20:57.066107 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.067448 11974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:20:57.067582 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "5218701608d04deca1175445803d1a10"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "52d78e967d3869aee95df7bc2c066e5c"
server_key_iv: "69bf6ec15c3638d6c8c163d822411e3c"
server_key_version: "encryptionkey@0"
I20260929 16:20:57.068123 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:57.089708 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:57.093364 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:57.093567 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:57.094192 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:57.095144 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:57.095216 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.095283 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:57.095332 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.106732 11974 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:43115
I20260929 16:20:57.106870 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:43115 every 8 connection(s)
I20260929 16:20:57.107962 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:20:57.113979 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 11974
I20260929 16:20:57.114181 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11331
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
I20260929 16:20:57.122957 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:39947
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:33163
--webserver_port=42829
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:20:57.128954 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.110118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:59435 (local address 127.4.200.126:33163)
0929 16:20:57.110267 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.110273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.111394 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 16:20:57.112832 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.112835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.112887 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.112983 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.115204 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.116464 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118241 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.118614 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118703 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.123483 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.123487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.123489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.123524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.125738 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.126818 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.126822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.126824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.126944 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127454 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.127457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.127459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.127702 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.127774 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.128634 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.128778 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:20:57.129130 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.111989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:40573 (local address 127.4.200.125:39541)
0929 16:20:57.112136 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.112147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.112285 (+ 138us) server_negotiation.cc:407] Connection header received
0929 16:20:57.113092 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.113096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.113145 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.113227 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.116154 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.117200 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118383 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.118728 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118800 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.123229 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.123236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.123242 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.123281 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.125918 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.126720 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.126727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.126732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.126942 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127468 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.127475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.127479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.127779 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.127884 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.128667 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.128841 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:20:57.130033 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.110663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:59435)
0929 16:20:57.111214 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.111273 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.112460 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.113114 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.113123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.113702 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.114980 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.115007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.116660 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.116663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.118092 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.118104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.118940 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.119260 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.120274 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.120291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.123312 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.125908 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.125917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.125941 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.126574 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.127062 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.127068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.127331 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.128147 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.128149 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.128348 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.128565 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.128940 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":728,"spinlock_wait_cycles":63360,"thread_start_us":175,"threads_started":1}
I20260929 16:20:57.130478 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.111838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.4.200.66:40573)
0929 16:20:57.112170 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.112185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.112460 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.113274 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.113277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.113702 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.115643 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.115656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.117380 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.117384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.118222 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.118230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.118947 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.119260 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.120372 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.120388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.122984 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.126075 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.126081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.126085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.126577 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.127125 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.127138 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.127340 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.127972 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.127993 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.128363 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.129560 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.129777 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":230,"spinlock_wait_cycles":23040,"thread_start_us":113,"threads_started":1}
I20260929 16:20:57.130990 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.110663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.4.200.66:43851)
0929 16:20:57.111215 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.111273 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.112480 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.113114 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.113124 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.113704 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.114982 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.115007 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.116363 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.116370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.118085 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.118099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.119085 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.119092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.119272 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.120208 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.120235 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.123102 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.126116 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.126119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.126121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.126574 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.127062 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.127068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.127331 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.130451 (+ 3120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.130454 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.130557 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.130660 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.130829 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":143,"spinlock_wait_cycles":36224,"thread_start_us":139,"threads_started":1}
I20260929 16:20:57.131240 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.110117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:43851 (local address 127.4.200.124:37523)
0929 16:20:57.110277 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.110283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.111399 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 16:20:57.112737 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.112742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.112820 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.112917 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.115193 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.116158 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118237 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.118781 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.118883 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.123305 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.123309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.123312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.123345 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.125989 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.126818 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.126822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.126824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.126944 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.127454 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.127457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.127459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.127991 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.128099 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.130793 (+ 2694us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.130993 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:57.132011 12105 heartbeater.cc:347] Connected to a master server at 127.4.200.125:39541
I20260929 16:20:57.131951 12103 heartbeater.cc:347] Connected to a master server at 127.4.200.124:37523
I20260929 16:20:57.132071 12104 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:57.132339 12103 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.132382 12104 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.132339 12105 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.133040 12104 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
I20260929 16:20:57.133044 12103 heartbeater.cc:511] Master 127.4.200.124:37523 requested a full tablet report, sending...
I20260929 16:20:57.133199 12105 heartbeater.cc:511] Master 127.4.200.125:39541 requested a full tablet report, sending...
I20260929 16:20:57.134545 11787 ts_manager.cc:194] Registered new tserver with Master: 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:57.134614 11604 ts_manager.cc:194] Registered new tserver with Master: 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:57.134756 11001 ts_manager.cc:194] Re-registered known tserver with Master: 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:57.135721 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:59435
I20260929 16:20:57.241597 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.236673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55240)
0929 16:20:57.236948 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.236964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.237101 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.237427 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.237430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.237450 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.237561 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.237567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.238643 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.238648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.239902 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.239912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.240871 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.240875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.241227 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.241256 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.241338 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.241429 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":60,"threads_started":1}
I20260929 16:20:57.241596 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.236771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.4.200.124:37523)
0929 16:20:57.236960 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.236965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.237066 (+ 101us) server_negotiation.cc:407] Connection header received
0929 16:20:57.237218 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.237222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.237288 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.237399 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.237675 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.238508 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.240083 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.240728 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.240849 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.241033 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.241374 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.241460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:20:57.242808 11797 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:20:57.243243 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128bfa733ff744bbbe6e105ec0ad0a62. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:57.243386 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: LEARNER
W20260929 16:20:57.247329 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:20:57.247596 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:20:57.247640 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:20:57.251777 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:20:57.251838 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:20:57.251900 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:20:57.257473 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:39947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=42829
--tserver_master_addrs=127.4.200.126:33163,127.4.200.125:39541,127.4.200.124:37523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:20:57.258569 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:20:57.259634 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:20:57.270227 12120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:57.270228 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:20:57.270227 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:20:57.270709 12110 server_base.cc:1034] running on GCE node
I20260929 16:20:57.271137 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:20:57.271832 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:20:57.273059 12110 hybrid_clock.cc:648] HybridClock initialized: now 1790698857273043 us; error 35 us; skew 500 ppm
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:20:57.275971 12110 init.cc:377] Logged in from keytab as oryx/127.4.200.67@KRBTEST.COM (short username oryx)
I20260929 16:20:57.276852 12110 webserver.cc:460] Webserver started at http://127.4.200.67:42829/ using document root <none> and password file <none>
I20260929 16:20:57.277422 12110 fs_manager.cc:359] Metadata directory not provided
I20260929 16:20:57.277554 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:20:57.281694 12110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:57.283895 12127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.285184 12110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:20:57.285333 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "5f3a71ad6ca142f3862c9e493424ffd2"
format_stamp: "Formatted at 2026-09-29 16:20:55 on dist-test-slave-s9ct"
server_key: "bd40609a5e295fae64a3377538b8d434"
server_key_iv: "0b07a6d865eddd30744e60ae8da04831"
server_key_version: "encryptionkey@0"
I20260929 16:20:57.285893 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:20:57.307797 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:20:57.310949 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:20:57.311167 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:20:57.311733 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:20:57.312534 12110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:20:57.312580 12110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.312623 12110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:20:57.312647 12110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:20:57.322270 12110 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:39947
I20260929 16:20:57.322304 12240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:39947 every 8 connection(s)
I20260929 16:20:57.323355 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:20:57.331670 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 12110
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.124@KRBTEST.COM
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.125@KRBTEST.COM
Sep 29 16:20:57 dist-test-slave-s9ct krb5kdc[10956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:20:57.341450 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.325384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:59419 (local address 127.4.200.124:37523)
0929 16:20:57.325617 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.325620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.326977 (+ 1357us) server_negotiation.cc:407] Connection header received
0929 16:20:57.327817 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.327823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.327890 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.327946 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.329504 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330314 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.331867 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.332202 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332274 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.336507 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.336511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.336514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.336546 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.338370 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339075 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.339080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.339085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.339187 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339605 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.339611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.339614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.339842 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.339922 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.341179 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.341296 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 16:20:57.342182 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.327580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.4.200.67:34907)
0929 16:20:57.327841 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.327855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.327951 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.328352 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.328360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.328797 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.329340 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.329355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330168 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.330174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.331491 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.331501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.332238 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.332486 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.333352 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.333363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.336687 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.339029 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.339039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.339288 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.339570 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.339581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.339680 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.340103 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.340106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.340479 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.340704 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.341074 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":90,"thread_start_us":109,"threads_started":1}
I20260929 16:20:57.342506 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.325384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:55691 (local address 127.4.200.126:33163)
0929 16:20:57.325527 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.325531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.326755 (+ 1224us) server_negotiation.cc:407] Connection header received
0929 16:20:57.327875 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.327878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.327934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.327984 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.329505 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330309 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.331654 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.331979 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332074 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.338025 (+ 5951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.338029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.338032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.338061 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.339811 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.340644 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.340647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.340650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.340741 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.341086 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.341090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.341092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.341301 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.341384 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.342145 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.342277 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:20:57.342790 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.325978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.4.200.67:59419)
0929 16:20:57.326891 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.326904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.327555 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.328355 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.328362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.328828 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.329354 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.329361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330875 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.330880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.331747 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.331755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.332433 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.332530 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.333318 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.333342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.336316 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.338541 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.338549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.338568 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.338961 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.339305 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.339315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.339498 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.339988 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.339995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.340984 (+ 989us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.341928 (+ 944us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.342083 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":738,"spinlock_wait_cycles":104320,"thread_start_us":599,"threads_started":1}
I20260929 16:20:57.342950 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.327683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:34907 (local address 127.4.200.125:39541)
0929 16:20:57.327897 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.327904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.327921 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:20:57.328071 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.328074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.328122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.328210 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.329490 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330025 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.331627 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.331966 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332036 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.336847 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.336850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.336853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.336879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.338902 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339391 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.339394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.339395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.339468 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339784 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.339787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.339788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.339988 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.340067 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.342631 (+ 2564us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.342780 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 16:20:57.343667 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.325978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:55691)
0929 16:20:57.326564 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.326625 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.327555 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.328355 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.328362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.328822 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:20:57.329352 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.329361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.330442 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.330445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.331483 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.331496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.332291 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.332295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.332478 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.333496 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:20:57.333511 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:20:57.337873 (+ 4362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:20:57.339946 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.339950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.339952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.340500 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.340870 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:20:57.340874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:20:57.340877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:20:57.340975 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:20:57.341443 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:20:57.341445 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:20:57.341929 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.343329 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.343482 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":148,"spinlock_wait_cycles":32640,"thread_start_us":152,"threads_started":1}
I20260929 16:20:57.343878 12243 heartbeater.cc:347] Connected to a master server at 127.4.200.125:39541
I20260929 16:20:57.344133 12241 heartbeater.cc:347] Connected to a master server at 127.4.200.124:37523
I20260929 16:20:57.344254 12241 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.344147 12243 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.344638 12242 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33163
I20260929 16:20:57.344704 12242 heartbeater.cc:464] Registering TS with master...
I20260929 16:20:57.344858 12242 heartbeater.cc:511] Master 127.4.200.126:33163 requested a full tablet report, sending...
I20260929 16:20:57.344844 12241 heartbeater.cc:511] Master 127.4.200.124:37523 requested a full tablet report, sending...
I20260929 16:20:57.344844 12243 heartbeater.cc:511] Master 127.4.200.125:39541 requested a full tablet report, sending...
I20260929 16:20:57.345952 11001 ts_manager.cc:194] Re-registered known tserver with Master: 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:57.345989 11787 ts_manager.cc:194] Registered new tserver with Master: 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:57.346480 11604 ts_manager.cc:194] Registered new tserver with Master: 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:57.346673 11001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:55691
I20260929 16:20:57.350889 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.334894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.4.200.125:39541)
0929 16:20:57.335161 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.335168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.335603 (+ 435us) server_negotiation.cc:407] Connection header received
0929 16:20:57.335769 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.335772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.335827 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.335889 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.336859 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.337718 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.339070 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.339475 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.339565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.341403 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.341407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.341409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.341434 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.343271 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.349185 (+ 5914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.349188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.349190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.349315 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.349662 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.349666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.349668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.350181 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.350324 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.350579 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.350710 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260929 16:20:57.352094 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.334086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.4.200.126:33163)
0929 16:20:57.335441 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.335446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.335460 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:57.335498 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.335501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.335559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.335685 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.336389 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.340812 (+ 4423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.344814 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.345165 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.345274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.347974 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.347977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.347979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.348000 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.349701 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.350166 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.350169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.350172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.350263 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.350621 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.350624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.350626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.351184 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.351320 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.351808 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.351935 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1256,"thread_start_us":80,"threads_started":1}
I20260929 16:20:57.352267 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.334785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.4.200.124:37523)
0929 16:20:57.341622 (+ 6837us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.341627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.341641 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:57.341674 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.341677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.341716 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.341789 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.345409 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.346055 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.347144 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.347425 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.347486 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.349001 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.349008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.349014 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.349042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.350646 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.351100 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.351103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.351105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.351183 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.351490 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.351495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.351497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.351707 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.351841 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.351983 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.352079 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6778,"thread_start_us":71,"threads_started":1}
I20260929 16:20:57.354933 11001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:57.357936 11001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:57.360728 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: NON_VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: true } } }
I20260929 16:20:57.360913 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:57.364357 12259 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: attempting to promote NON_VOTER 6ec00ccb6715431894a52d2d24ba4b89 to VOTER
I20260929 16:20:57.364773 12259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } }
I20260929 16:20:57.365918 11614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Refusing update from remote peer 128bfa733ff744bbbe6e105ec0ad0a62: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:20:57.365969 11797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 LEARNER]: Refusing update from remote peer 128bfa733ff744bbbe6e105ec0ad0a62: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:20:57.366289 12111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:20:57.366433 12259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:20:57.368249 12261 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.368659 11614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.368781 11797 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.369835 12260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.369939 12260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:57.369896 12261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.370007 12261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:57.370081 12260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.370172 12260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:57.371376 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.371520 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:57.371975 12269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:57.372108 12269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:57.381294 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.375754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44163 (local address 127.0.0.1:39758)
0929 16:20:57.375906 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.375922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.376001 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.376782 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.376785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.376802 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.376910 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.376917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.378483 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.378487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.379662 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.379672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380807 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.380810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.380923 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.380972 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.381050 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.381119 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":72}
I20260929 16:20:57.381325 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.376042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:43115 (local address 127.0.0.1:41596)
0929 16:20:57.376531 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.376544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.376627 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.377374 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.377379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.377398 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.377501 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.377507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.378827 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.378833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.379912 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.379920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380933 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.380937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.381054 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.381076 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.381125 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.381187 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260929 16:20:57.381335 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.375896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.4.200.65:44163)
0929 16:20:57.376241 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.376258 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.376284 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:20:57.376367 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.376375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.376622 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.377058 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.377191 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.378351 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.379793 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380691 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.380804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.380972 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.381086 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.381185 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":83,"threads_started":1}
I20260929 16:20:57.381803 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.376413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.4.200.66:43115)
0929 16:20:57.376864 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.376875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.376896 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:20:57.376967 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.376974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.377195 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.377383 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.377615 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.378702 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.380031 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380806 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.381462 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.381597 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.381637 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.381703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":100,"threads_started":1}
I20260929 16:20:57.383960 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.376322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:39947 (local address 127.0.0.1:48920)
0929 16:20:57.378610 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.378622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.378707 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.379283 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.379286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.379302 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.379378 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.379383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380625 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.380638 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.381789 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.381798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.383568 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.383572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.383663 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.383686 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.383767 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.383836 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2217,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260929 16:20:57.384038 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.376709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.4.200.67:39947)
0929 16:20:57.377007 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.377018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.378725 (+ 1707us) server_negotiation.cc:407] Connection header received
0929 16:20:57.378807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.378812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.379003 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.379154 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.379489 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.380519 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.381929 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.382752 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.382865 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.382984 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.383815 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.383907 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":128,"threads_started":1}
I20260929 16:20:57.384538 11900 tablet_service.cc:1467] Processing CreateTablet for tablet 7d498049ae8947f5933fdd17eb561b63 (DEFAULT_TABLE table=test-table [id=142126ffc65f4be49cae0ee502f81bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:57.384526 12037 tablet_service.cc:1467] Processing CreateTablet for tablet 7d498049ae8947f5933fdd17eb561b63 (DEFAULT_TABLE table=test-table [id=142126ffc65f4be49cae0ee502f81bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:57.385486 11900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d498049ae8947f5933fdd17eb561b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:57.385504 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d498049ae8947f5933fdd17eb561b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:57.386793 12175 tablet_service.cc:1467] Processing CreateTablet for tablet 7d498049ae8947f5933fdd17eb561b63 (DEFAULT_TABLE table=test-table [id=142126ffc65f4be49cae0ee502f81bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:57.387674 12175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d498049ae8947f5933fdd17eb561b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:57.392515 12275 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Bootstrap starting.
I20260929 16:20:57.393591 12276 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Bootstrap starting.
I20260929 16:20:57.394907 12275 tablet_bootstrap.cc:654] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:57.395134 12277 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Bootstrap starting.
I20260929 16:20:57.395675 12276 tablet_bootstrap.cc:654] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:57.395905 12275 log.cc:824] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:57.396956 12276 log.cc:824] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:57.397601 12277 tablet_bootstrap.cc:654] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:57.398290 12277 log.cc:824] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:20:57.398857 12276 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: No bootstrap required, opened a new log
I20260929 16:20:57.399075 12276 ts_tablet_manager.cc:1397] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:20:57.400009 12277 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: No bootstrap required, opened a new log
I20260929 16:20:57.400209 12277 ts_tablet_manager.cc:1397] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:20:57.400236 12275 tablet_bootstrap.cc:492] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: No bootstrap required, opened a new log
I20260929 16:20:57.400394 12275 ts_tablet_manager.cc:1397] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:20:57.403359 12277 raft_consensus.cc:348] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.403590 12277 raft_consensus.cc:374] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:57.403650 12277 raft_consensus.cc:729] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3a71ad6ca142f3862c9e493424ffd2, State: Initialized, Role: FOLLOWER
I20260929 16:20:57.403857 12276 raft_consensus.cc:348] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.404228 12276 raft_consensus.cc:374] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:57.404179 12277 consensus_queue.cc:260] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.404309 12276 raft_consensus.cc:729] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5218701608d04deca1175445803d1a10, State: Initialized, Role: FOLLOWER
I20260929 16:20:57.404883 12276 consensus_queue.cc:260] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.405334 12242 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
I20260929 16:20:57.405331 12277 ts_tablet_manager.cc:1428] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:20:57.405308 12275 raft_consensus.cc:348] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.405534 12275 raft_consensus.cc:374] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:57.405581 12275 raft_consensus.cc:729] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc685fd96de045cf8557c575cf51477a, State: Initialized, Role: FOLLOWER
I20260929 16:20:57.405936 12276 ts_tablet_manager.cc:1428] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 16:20:57.406071 12275 consensus_queue.cc:260] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.406982 12275 ts_tablet_manager.cc:1428] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:20:57.407256 12104 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
I20260929 16:20:57.408396 11967 heartbeater.cc:503] Master 127.4.200.126:33163 was elected leader, sending a full tablet report...
W20260929 16:20:57.575017 12244 tablet.cc:2369] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:20:57.609853 12106 tablet.cc:2369] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:57.616062 12281 raft_consensus.cc:482] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:57.616310 12281 raft_consensus.cc:504] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.617447 12281 leader_election.cc:288] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163), 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:57.621403 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.617843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41899 (local address 127.4.200.65:44163)
0929 16:20:57.618017 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.618023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.618043 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:20:57.618107 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.618110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.618163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.618254 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.618608 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.619281 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.620466 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.620863 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.620994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.621059 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.621217 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.621283 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:20:57.621411 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.617746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44163 (local address 127.4.200.67:41899)
0929 16:20:57.617894 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.617911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.618006 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.618281 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.618284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.618303 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.618476 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.618482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.619386 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.619392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.620347 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.620356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.620984 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.620987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.621090 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.621118 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.621201 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.621273 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:20:57.621505 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.618044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:43115 (local address 127.4.200.67:44779)
0929 16:20:57.618175 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:57.618190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:57.618257 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:57.618556 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:57.618559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:57.618573 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:57.618691 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.618697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.619559 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.619565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.620411 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:57.620420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.621070 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.621074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:57.621188 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.621204 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:20:57.621292 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:20:57.621362 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:20:57.621505 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.618141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:44779 (local address 127.4.200.66:43115)
0929 16:20:57.618293 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.618300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.618318 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:20:57.618371 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.618373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.618452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.618547 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:57.618830 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.619442 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.620529 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.620948 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.621079 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.621127 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.621307 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.621375 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:20:57.622223 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d498049ae8947f5933fdd17eb561b63" candidate_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5218701608d04deca1175445803d1a10" is_pre_election: true
I20260929 16:20:57.622232 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d498049ae8947f5933fdd17eb561b63" candidate_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc685fd96de045cf8557c575cf51477a" is_pre_election: true
I20260929 16:20:57.622534 12055 raft_consensus.cc:2454] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3a71ad6ca142f3862c9e493424ffd2 in term 0.
I20260929 16:20:57.622653 11920 raft_consensus.cc:2454] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3a71ad6ca142f3862c9e493424ffd2 in term 0.
I20260929 16:20:57.623061 12128 leader_election.cc:302] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5218701608d04deca1175445803d1a10, 5f3a71ad6ca142f3862c9e493424ffd2; no voters:
I20260929 16:20:57.623337 12281 raft_consensus.cc:2781] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:57.623430 12281 raft_consensus.cc:482] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:57.623486 12281 raft_consensus.cc:3037] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:57.624557 12281 raft_consensus.cc:504] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.624886 12281 leader_election.cc:288] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [CANDIDATE]: Term 1 election: Requested vote from peers dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163), 5218701608d04deca1175445803d1a10 (127.4.200.66:43115)
I20260929 16:20:57.625233 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d498049ae8947f5933fdd17eb561b63" candidate_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc685fd96de045cf8557c575cf51477a"
I20260929 16:20:57.625280 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d498049ae8947f5933fdd17eb561b63" candidate_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5218701608d04deca1175445803d1a10"
I20260929 16:20:57.625403 11920 raft_consensus.cc:3037] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:57.625417 12055 raft_consensus.cc:3037] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:57.626339 11920 raft_consensus.cc:2454] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3a71ad6ca142f3862c9e493424ffd2 in term 1.
I20260929 16:20:57.626673 12128 leader_election.cc:302] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3a71ad6ca142f3862c9e493424ffd2, dc685fd96de045cf8557c575cf51477a; no voters:
I20260929 16:20:57.626857 12281 raft_consensus.cc:2781] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:57.627024 12055 raft_consensus.cc:2454] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3a71ad6ca142f3862c9e493424ffd2 in term 1.
I20260929 16:20:57.627090 12281 raft_consensus.cc:686] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 LEADER]: Becoming Leader. State: Replica: 5f3a71ad6ca142f3862c9e493424ffd2, State: Running, Role: LEADER
I20260929 16:20:57.627730 12281 consensus_queue.cc:237] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } }
I20260929 16:20:57.631095 11000 catalog_manager.cc:5495] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67). New cstate: current_term: 1 leader_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } health_report { overall_health: HEALTHY } } }
W20260929 16:20:57.651705 11969 tablet.cc:2369] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:20:57.673481 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.670134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.4.200.67:39947)
0929 16:20:57.670257 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.670260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.670274 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:57.670435 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.670437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.670491 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.670580 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:57.670823 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.671347 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.672213 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.672411 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.672483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.672764 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:57.673156 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:57.673267 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.673328 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.673376 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:20:57.679092 12055 raft_consensus.cc:1261] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Refusing update from remote peer 5f3a71ad6ca142f3862c9e493424ffd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:57.679216 11920 raft_consensus.cc:1261] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Refusing update from remote peer 5f3a71ad6ca142f3862c9e493424ffd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:57.679831 12281 consensus_queue.cc:1035] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:57.680016 12284 consensus_queue.cc:1035] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:57.687850 12287 mvcc.cc:205] Tried to move back new op lower bound from 7334702521046433792 to 7334702520845352960. Current Snapshot: MvccSnapshot[applied={T|T < 7334702521046433792}]
I20260929 16:20:57.688721 12289 mvcc.cc:205] Tried to move back new op lower bound from 7334702521046433792 to 7334702520845352960. Current Snapshot: MvccSnapshot[applied={T|T < 7334702521046433792}]
I20260929 16:20:57.693013 12288 mvcc.cc:205] Tried to move back new op lower bound from 7334702521046433792 to 7334702520845352960. Current Snapshot: MvccSnapshot[applied={T|T < 7334702521046433792}]
I20260929 16:20:57.698884 11001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:57.699010 11001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:57.703047 11001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 128bfa733ff744bbbe6e105ec0ad0a62: Sending DeleteTablet for 3 replicas of tablet 7d498049ae8947f5933fdd17eb561b63
I20260929 16:20:57.703902 12175 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d498049ae8947f5933fdd17eb561b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:57 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:48920
I20260929 16:20:57.705380 11900 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d498049ae8947f5933fdd17eb561b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:57 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:39758
I20260929 16:20:57.705850 12298 tablet_replica.cc:330] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: stopping tablet replica
I20260929 16:20:57.706120 12037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d498049ae8947f5933fdd17eb561b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:57 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:41596
I20260929 16:20:57.706176 12298 raft_consensus.cc:2229] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:57.706523 12298 raft_consensus.cc:2258] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:57.707365 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 10969
I20260929 16:20:57.708650 12298 ts_tablet_manager.cc:1905] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:57.711679 12298 ts_tablet_manager.cc:1918] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:57.711860 12298 log.cc:1196] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/7d498049ae8947f5933fdd17eb561b63
I20260929 16:20:57.712301 12298 ts_tablet_manager.cc:1939] T 7d498049ae8947f5933fdd17eb561b63 P dc685fd96de045cf8557c575cf51477a: Deleting consensus metadata
W20260929 16:20:57.714287 11854 connection.cc:535] server connection from 127.0.0.1:39758 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:20:57.715088 12299 tablet_replica.cc:330] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: stopping tablet replica
I20260929 16:20:57.715420 12299 raft_consensus.cc:2229] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:57.715749 12299 raft_consensus.cc:2258] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:57.717756 12299 ts_tablet_manager.cc:1905] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:57.718621 12296 tablet_replica.cc:330] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: stopping tablet replica
I20260929 16:20:57.718907 12296 raft_consensus.cc:2229] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:57.719393 12296 raft_consensus.cc:2258] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:57.720570 12299 ts_tablet_manager.cc:1918] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:57.720675 12299 log.cc:1196] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/7d498049ae8947f5933fdd17eb561b63
I20260929 16:20:57.720978 12299 ts_tablet_manager.cc:1939] T 7d498049ae8947f5933fdd17eb561b63 P 5218701608d04deca1175445803d1a10: Deleting consensus metadata
I20260929 16:20:57.720965 12296 ts_tablet_manager.cc:1905] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:20:57.722054 11993 connection.cc:441] server connection from 127.0.0.1:41596 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41596 (request call id 1) could send its response
I20260929 16:20:57.723982 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.705020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.4.200.125:39541)
0929 16:20:57.705181 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.705186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.705657 (+ 471us) server_negotiation.cc:407] Connection header received
0929 16:20:57.705955 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.705958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.706005 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.706053 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.714711 (+ 8658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.715519 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.716729 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.717100 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.717220 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.719640 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.719644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.719648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.719678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.721755 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.722471 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.722477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.722478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.722574 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.722909 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.722913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.722914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.723136 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.723257 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.723668 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.723800 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:20:57.724378 12296 ts_tablet_manager.cc:1918] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:57.724473 12296 log.cc:1196] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/7d498049ae8947f5933fdd17eb561b63
I20260929 16:20:57.724781 12296 ts_tablet_manager.cc:1939] T 7d498049ae8947f5933fdd17eb561b63 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting consensus metadata
W20260929 16:20:57.726909 12131 connection.cc:441] server connection from 127.0.0.1:48920 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48920 (request call id 1) could send its response
I20260929 16:20:57.732519 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.717980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.4.200.125:39541)
0929 16:20:57.718085 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.718088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.723082 (+ 4994us) server_negotiation.cc:407] Connection header received
0929 16:20:57.723267 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.723273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.723321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.723372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.724237 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.724996 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.726530 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.726834 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.726914 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.728462 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.728468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.728473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.728500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.730316 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.730818 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.730822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.730824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.730909 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.731239 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.731243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.731245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.731470 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.731607 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.732207 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.732323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:20:57.734961 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.705315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.4.200.124:37523)
0929 16:20:57.705409 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.705413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.725049 (+ 19636us) server_negotiation.cc:407] Connection header received
0929 16:20:57.725240 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.725244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.725296 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.725353 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.726287 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.727124 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.728283 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.728619 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.728728 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.730657 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.730660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.730664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.730697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.732744 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.733455 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.733461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.733466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.733553 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.733853 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.733857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.733859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.734112 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.734246 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.734657 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.734781 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
I20260929 16:20:57.737308 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:57.722416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.4.200.124:37523)
0929 16:20:57.722567 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:57.722572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:57.727622 (+ 5050us) server_negotiation.cc:407] Connection header received
0929 16:20:57.728551 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:57.728554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:57.728602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:57.728676 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:20:57.729459 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.730219 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.731526 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:57.731897 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:57.731978 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:57.733804 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:20:57.733808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:20:57.733814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:20:57.733843 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:20:57.735606 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.736093 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.736095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.736096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.736171 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:20:57.736475 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:20:57.736479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:20:57.736483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:20:57.736702 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:20:57.736800 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:57.737053 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:20:57.737149 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":48}
I20260929 16:20:58.694998 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:58.694497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:32865)
0929 16:20:58.694774 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:58.694880 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
W20260929 16:20:58.695334 11967 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:33163 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:58.695664 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:58.694827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:43585)
0929 16:20:58.695483 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:58.695575 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":62,"threads_started":1}
I20260929 16:20:58.695814 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:58.695322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:36997)
0929 16:20:58.695633 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:58.695722 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
W20260929 16:20:58.695940 12104 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:33163 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
W20260929 16:20:58.696055 12242 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:33163 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.203773 12310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 128bfa733ff744bbbe6e105ec0ad0a62)
I20260929 16:20:59.203914 12310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } }
I20260929 16:20:59.204378 12310 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163), 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125:39541)
I20260929 16:20:59.204998 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.204578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37868)
0929 16:20:59.204787 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.204881 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
W20260929 16:20:59.205237 11773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.206061 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.205842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37884)
0929 16:20:59.205950 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.206004 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:20:59.206211 11773 leader_election.cc:334] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163): Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.207302 12315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 128bfa733ff744bbbe6e105ec0ad0a62)
I20260929 16:20:59.207481 12315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } }
I20260929 16:20:59.208453 12315 leader_election.cc:288] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163), 6ec00ccb6715431894a52d2d24ba4b89 (127.4.200.124:37523)
I20260929 16:20:59.208775 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.204906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:39541 (local address 127.0.0.1:49430)
0929 16:20:59.205147 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.205165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.205275 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.205583 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.205586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.205604 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.205718 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.205724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.206704 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.206711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.207646 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.207656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.208311 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.208315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.208411 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.208433 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.208527 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.208618 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260929 16:20:59.208876 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.208505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37890)
0929 16:20:59.208706 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.208779 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
W20260929 16:20:59.209066 11590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.209226 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.205062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.4.200.125:39541)
0929 16:20:59.205360 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.205364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.205378 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:59.205434 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.205436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.205473 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.205572 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.205832 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.206594 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.207769 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.208185 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.208300 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.208361 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.208977 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.209068 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":71,"threads_started":1}
I20260929 16:20:59.209643 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec00ccb6715431894a52d2d24ba4b89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "97e744cba70f426b9666d9d85be8c1c2" is_pre_election: true
I20260929 16:20:59.209807 11614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec00ccb6715431894a52d2d24ba4b89 in term 1.
I20260929 16:20:59.209969 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.209703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37894)
0929 16:20:59.209856 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.209910 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:20:59.210104 11773 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ec00ccb6715431894a52d2d24ba4b89, 97e744cba70f426b9666d9d85be8c1c2; no voters: 128bfa733ff744bbbe6e105ec0ad0a62
W20260929 16:20:59.210222 11590 leader_election.cc:334] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163): Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.210336 12310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:20:59.210422 12310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 128bfa733ff744bbbe6e105ec0ad0a62)
I20260929 16:20:59.210479 12310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:20:59.211609 12310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } }
I20260929 16:20:59.211979 12310 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 election: Requested vote from peers 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163), 97e744cba70f426b9666d9d85be8c1c2 (127.4.200.125:39541)
I20260929 16:20:59.212311 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.212072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37904)
0929 16:20:59.212159 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.212236 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 16:20:59.212447 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec00ccb6715431894a52d2d24ba4b89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "97e744cba70f426b9666d9d85be8c1c2"
I20260929 16:20:59.212572 11614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:20:59.212988 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.212775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37916)
0929 16:20:59.212881 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.212930 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:20:59.213068 11773 leader_election.cc:334] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163): Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
I20260929 16:20:59.213130 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.208645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:37523 (local address 127.0.0.1:55276)
0929 16:20:59.208796 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.208808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.208898 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.209923 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.209926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.209945 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.210053 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.210062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.210931 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.210936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.211837 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.211848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.212725 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.212732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.212840 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.212864 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.212946 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.213016 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260929 16:20:59.213394 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.209062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.4.200.124:37523)
0929 16:20:59.209676 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.209686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.209698 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:20:59.209737 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.209740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.209804 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.209900 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.210162 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.210819 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.211965 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.212496 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.212603 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.212666 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.212957 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.213034 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":93,"threads_started":1}
I20260929 16:20:59.213768 11614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec00ccb6715431894a52d2d24ba4b89 in term 2.
I20260929 16:20:59.213769 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97e744cba70f426b9666d9d85be8c1c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6ec00ccb6715431894a52d2d24ba4b89" is_pre_election: true
I20260929 16:20:59.213984 11797 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97e744cba70f426b9666d9d85be8c1c2 in current term 2: Already voted for candidate 6ec00ccb6715431894a52d2d24ba4b89 in this term.
I20260929 16:20:59.214064 11773 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ec00ccb6715431894a52d2d24ba4b89, 97e744cba70f426b9666d9d85be8c1c2; no voters: 128bfa733ff744bbbe6e105ec0ad0a62
I20260929 16:20:59.214341 12310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:20:59.214505 11589 leader_election.cc:302] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97e744cba70f426b9666d9d85be8c1c2; no voters: 128bfa733ff744bbbe6e105ec0ad0a62, 6ec00ccb6715431894a52d2d24ba4b89
I20260929 16:20:59.214594 12310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [term 2 LEADER]: Becoming Leader. State: Replica: 6ec00ccb6715431894a52d2d24ba4b89, State: Running, Role: LEADER
I20260929 16:20:59.214865 12310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } }
I20260929 16:20:59.214923 12315 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:20:59.216308 12320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec00ccb6715431894a52d2d24ba4b89. Latest consensus state: current_term: 2 leader_uuid: "6ec00ccb6715431894a52d2d24ba4b89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:59.216442 12320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:59.216825 12322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:20:59.219216 12322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:20:59.219653 12322 catalog_manager.cc:1259] Loaded cluster ID: 2ae92c7d808b46869257d3a8ee9603f6
I20260929 16:20:59.219732 12322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:20:59.220451 12322 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:20:59.221662 12322 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Loaded TSK: 0
I20260929 16:20:59.222882 12322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:20:59.258278 11786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:20:59.261503 11786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:20:59.264788 11614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [term 2 FOLLOWER]: Refusing update from remote peer 6ec00ccb6715431894a52d2d24ba4b89: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 16:20:59.264848 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.264579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37934)
0929 16:20:59.264726 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.264780 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:20:59.265331 12310 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 16:20:59.265702 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.265429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.0.0.1:37948)
0929 16:20:59.265602 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.265651 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 16:20:59.265864 11773 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 -> Peer 128bfa733ff744bbbe6e105ec0ad0a62 (127.4.200.126:33163): Couldn't send request to peer 128bfa733ff744bbbe6e105ec0ad0a62. Status: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:20:59.266947 12315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec00ccb6715431894a52d2d24ba4b89. Latest consensus state: current_term: 2 leader_uuid: "6ec00ccb6715431894a52d2d24ba4b89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:59.267093 12315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:59.268215 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6ec00ccb6715431894a52d2d24ba4b89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:59.268350 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89 [sys.catalog]: This master's current role is: LEADER
I20260929 16:20:59.268460 12315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6ec00ccb6715431894a52d2d24ba4b89" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "128bfa733ff744bbbe6e105ec0ad0a62" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33163 } } peers { permanent_uuid: "97e744cba70f426b9666d9d85be8c1c2" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 39541 } attrs { promote: false } } peers { permanent_uuid: "6ec00ccb6715431894a52d2d24ba4b89" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 37523 } attrs { promote: false } } }
I20260929 16:20:59.268599 12315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e744cba70f426b9666d9d85be8c1c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:20:59.268857 12324 mvcc.cc:205] Tried to move back new op lower bound from 7334702527543218176 to 7334702527346511872. Current Snapshot: MvccSnapshot[applied={T|T < 7334702527543218176}]
I20260929 16:20:59.279047 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.274278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44163 (local address 127.0.0.1:39772)
0929 16:20:59.274735 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.274751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.274845 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.275134 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.275136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.275158 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.275266 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.275277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.276309 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.276317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.277556 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.277564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.278622 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.278626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.278729 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.278758 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.278807 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.278888 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":27}
I20260929 16:20:59.279506 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.275553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:43115 (local address 127.0.0.1:41604)
0929 16:20:59.275786 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.275796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.275860 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.276247 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.276250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.276264 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.276346 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.276353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.277148 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.277151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.278210 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.278218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.279023 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.279027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.279252 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.279273 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.279343 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.279412 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"thread_start_us":125,"threads_started":1}
I20260929 16:20:59.279601 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.274557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.4.200.65:44163)
0929 16:20:59.274824 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.274828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.274842 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:20:59.274940 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.274944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.274992 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.275121 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.275413 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.276179 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.277687 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.278482 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.279296 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.279364 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.279400 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.279471 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260929 16:20:59.279933 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.275848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.4.200.66:43115)
0929 16:20:59.276047 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.276050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.276061 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:20:59.276107 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.276110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.276145 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.276242 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.276468 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.277019 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.278328 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.278876 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.279053 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.279120 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.279730 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.279816 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260929 16:20:59.280108 11900 tablet_service.cc:1467] Processing CreateTablet for tablet f70366bad94f40ff82e890aa5a0bb769 (DEFAULT_TABLE table=test-table [id=806157163a9449d8aa32a5c6e4b51fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:59.280278 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.274626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:39947 (local address 127.0.0.1:48942)
0929 16:20:59.274988 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.275000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.275059 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.276374 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.276379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.276398 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.276472 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.276477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.277460 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.277463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.278741 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.278749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.279943 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.279947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.280026 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.280048 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.280097 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.280171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":7}
I20260929 16:20:59.280424 11900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70366bad94f40ff82e890aa5a0bb769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:59.280833 12037 tablet_service.cc:1467] Processing CreateTablet for tablet f70366bad94f40ff82e890aa5a0bb769 (DEFAULT_TABLE table=test-table [id=806157163a9449d8aa32a5c6e4b51fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:59.281075 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.274711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.4.200.67:39947)
0929 16:20:59.276149 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.276154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.276169 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:20:59.276221 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.276227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.276288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.276373 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.276583 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.277335 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.278896 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.279814 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.280771 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.280840 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.280881 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.280952 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":62,"threads_started":1}
I20260929 16:20:59.281103 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70366bad94f40ff82e890aa5a0bb769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:59.281540 12175 tablet_service.cc:1467] Processing CreateTablet for tablet f70366bad94f40ff82e890aa5a0bb769 (DEFAULT_TABLE table=test-table [id=806157163a9449d8aa32a5c6e4b51fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:20:59.282013 12175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70366bad94f40ff82e890aa5a0bb769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:20:59.283627 12333 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: Bootstrap starting.
I20260929 16:20:59.284400 12334 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Bootstrap starting.
I20260929 16:20:59.284528 12333 tablet_bootstrap.cc:654] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:59.285199 12335 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Bootstrap starting.
I20260929 16:20:59.285777 12334 tablet_bootstrap.cc:654] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:59.285810 12333 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: No bootstrap required, opened a new log
I20260929 16:20:59.285885 12333 ts_tablet_manager.cc:1397] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:20:59.286123 12335 tablet_bootstrap.cc:654] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Neither blocks nor log segments found. Creating new log.
I20260929 16:20:59.286422 12333 raft_consensus.cc:348] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.286563 12333 raft_consensus.cc:374] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:59.286604 12333 raft_consensus.cc:729] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc685fd96de045cf8557c575cf51477a, State: Initialized, Role: FOLLOWER
I20260929 16:20:59.286754 12333 consensus_queue.cc:260] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.287139 12333 ts_tablet_manager.cc:1428] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 16:20:59.287350 12335 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: No bootstrap required, opened a new log
I20260929 16:20:59.287405 12334 tablet_bootstrap.cc:492] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: No bootstrap required, opened a new log
I20260929 16:20:59.287436 12335 ts_tablet_manager.cc:1397] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:20:59.287487 12334 ts_tablet_manager.cc:1397] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:20:59.287817 12335 raft_consensus.cc:348] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.287935 12335 raft_consensus.cc:374] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:59.287974 12335 raft_consensus.cc:729] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3a71ad6ca142f3862c9e493424ffd2, State: Initialized, Role: FOLLOWER
I20260929 16:20:59.287978 12334 raft_consensus.cc:348] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.288103 12334 raft_consensus.cc:374] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:20:59.288143 12334 raft_consensus.cc:729] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5218701608d04deca1175445803d1a10, State: Initialized, Role: FOLLOWER
I20260929 16:20:59.288120 12335 consensus_queue.cc:260] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.288286 12334 consensus_queue.cc:260] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.288455 12335 ts_tablet_manager.cc:1428] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:20:59.288614 12334 ts_tablet_manager.cc:1428] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:20:59.290535 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.287919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:43831)
0929 16:20:59.290345 (+ 2426us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.290430 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2350,"thread_start_us":114,"threads_started":1}
I20260929 16:20:59.291289 11966 heartbeater.cc:503] Master 127.4.200.124:37523 was elected leader, sending a full tablet report...
I20260929 16:20:59.291314 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.290914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:40865)
0929 16:20:59.291186 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.291239 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
I20260929 16:20:59.292621 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.290914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:57355)
0929 16:20:59.292454 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.292521 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1474,"thread_start_us":96,"threads_started":1}
I20260929 16:20:59.293005 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.289330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:35935)
0929 16:20:59.292866 (+ 3536us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.292922 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3465,"thread_start_us":86,"threads_started":1}
I20260929 16:20:59.293315 12241 heartbeater.cc:503] Master 127.4.200.124:37523 was elected leader, sending a full tablet report...
I20260929 16:20:59.293722 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.293499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:47065)
0929 16:20:59.293604 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.293661 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 16:20:59.465184 12342 raft_consensus.cc:482] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:20:59.465337 12342 raft_consensus.cc:504] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.466491 12342 leader_election.cc:288] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163), 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:59.471165 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.466767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44163 (local address 127.4.200.66:43153)
0929 16:20:59.467147 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.467163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.467292 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.467568 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.467571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.467597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.467806 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.467812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.468866 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.468871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.469955 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.469966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.470704 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.470707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.470814 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.470832 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.470920 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.471046 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":33,"spinlock_wait_cycles":6144}
I20260929 16:20:59.471164 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.466884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:43153 (local address 127.4.200.65:44163)
0929 16:20:59.467027 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.467032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.467259 (+ 227us) server_negotiation.cc:407] Connection header received
0929 16:20:59.467402 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.467405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.467466 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.467543 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.467928 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.468738 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.470085 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.470565 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.470703 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.470761 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.470935 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.471025 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:20:59.471581 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.466962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:57663 (local address 127.4.200.67:39947)
0929 16:20:59.467106 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.467110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.467258 (+ 148us) server_negotiation.cc:407] Connection header received
0929 16:20:59.467404 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.467407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.467456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.467543 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:20:59.467928 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.468720 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.470510 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.470911 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.471048 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.471097 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.471371 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.471466 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:20:59.471606 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70366bad94f40ff82e890aa5a0bb769" candidate_uuid: "5218701608d04deca1175445803d1a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc685fd96de045cf8557c575cf51477a" is_pre_election: true
I20260929 16:20:59.471792 11920 raft_consensus.cc:2454] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5218701608d04deca1175445803d1a10 in term 0.
I20260929 16:20:59.472034 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.466787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:39947 (local address 127.4.200.66:57663)
0929 16:20:59.467147 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.467163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:20:59.467292 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:20:59.467568 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:20:59.467571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:20:59.467594 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:20:59.467805 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.467812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.468824 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.468827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.469957 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:20:59.469967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.471033 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.471036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:20:59.471148 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.471175 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:20:59.471755 (+ 580us) client_negotiation.cc:240] Negotiation successful
0929 16:20:59.471829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":145,"spinlock_wait_cycles":14848,"thread_start_us":132,"threads_started":1}
I20260929 16:20:59.472306 11990 leader_election.cc:302] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5218701608d04deca1175445803d1a10, dc685fd96de045cf8557c575cf51477a; no voters:
I20260929 16:20:59.472630 12342 raft_consensus.cc:2781] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:20:59.472573 12195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70366bad94f40ff82e890aa5a0bb769" candidate_uuid: "5218701608d04deca1175445803d1a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" is_pre_election: true
I20260929 16:20:59.472703 12342 raft_consensus.cc:482] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:20:59.472762 12342 raft_consensus.cc:3037] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:59.472823 12195 raft_consensus.cc:2454] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5218701608d04deca1175445803d1a10 in term 0.
I20260929 16:20:59.473887 12342 raft_consensus.cc:504] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.474274 12342 leader_election.cc:288] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [CANDIDATE]: Term 1 election: Requested vote from peers dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163), 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947)
I20260929 16:20:59.474721 12195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70366bad94f40ff82e890aa5a0bb769" candidate_uuid: "5218701608d04deca1175445803d1a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3a71ad6ca142f3862c9e493424ffd2"
I20260929 16:20:59.474756 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70366bad94f40ff82e890aa5a0bb769" candidate_uuid: "5218701608d04deca1175445803d1a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc685fd96de045cf8557c575cf51477a"
I20260929 16:20:59.474861 12195 raft_consensus.cc:3037] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:59.474875 11920 raft_consensus.cc:3037] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:20:59.476027 11920 raft_consensus.cc:2454] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5218701608d04deca1175445803d1a10 in term 1.
I20260929 16:20:59.476019 12195 raft_consensus.cc:2454] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5218701608d04deca1175445803d1a10 in term 1.
I20260929 16:20:59.476313 11993 leader_election.cc:302] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5218701608d04deca1175445803d1a10, 5f3a71ad6ca142f3862c9e493424ffd2; no voters:
I20260929 16:20:59.476536 12342 raft_consensus.cc:2781] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:20:59.476783 12342 raft_consensus.cc:686] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 1 LEADER]: Becoming Leader. State: Replica: 5218701608d04deca1175445803d1a10, State: Running, Role: LEADER
I20260929 16:20:59.477051 12103 heartbeater.cc:503] Master 127.4.200.124:37523 was elected leader, sending a full tablet report...
I20260929 16:20:59.477075 12342 consensus_queue.cc:237] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } }
I20260929 16:20:59.478871 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.478530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:56581)
0929 16:20:59.478688 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.478745 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 16:20:59.479646 11786 catalog_manager.cc:5495] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5218701608d04deca1175445803d1a10 (127.4.200.66). New cstate: current_term: 1 leader_uuid: "5218701608d04deca1175445803d1a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5218701608d04deca1175445803d1a10" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 43115 } health_report { overall_health: HEALTHY } } }
I20260929 16:20:59.523177 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.519779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.4.200.66:43115)
0929 16:20:59.519975 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:20:59.519984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:20:59.520004 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:20:59.520153 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:20:59.520155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:20:59.520211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:20:59.520299 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:20:59.520503 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.521054 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.521828 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:20:59.522025 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:20:59.522099 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:20:59.522312 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:20:59.522809 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:20:59.522895 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:20:59.522960 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 16:20:59.523005 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:20:59.525823 11920 raft_consensus.cc:1261] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Refusing update from remote peer 5218701608d04deca1175445803d1a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:59.526145 12195 raft_consensus.cc:1261] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Refusing update from remote peer 5218701608d04deca1175445803d1a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:20:59.526376 12347 consensus_queue.cc:1035] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc685fd96de045cf8557c575cf51477a" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:59.526757 12342 consensus_queue.cc:1035] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3a71ad6ca142f3862c9e493424ffd2" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:20:59.528435 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.527989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:54203)
0929 16:20:59.528266 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.528329 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260929 16:20:59.529313 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.529055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:53989)
0929 16:20:59.529189 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.529245 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:20:59.531637 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.531403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.66:42269)
0929 16:20:59.531508 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.531559 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 16:20:59.532521 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.532272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.65:48819)
0929 16:20:59.532407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.532453 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:20:59.532846 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.532012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33163 (local address 127.4.200.67:48637)
0929 16:20:59.532711 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.532779 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:33163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633}
I20260929 16:20:59.539664 11786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:59.539835 11786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:20:59.543520 11786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ec00ccb6715431894a52d2d24ba4b89: Sending DeleteTablet for 3 replicas of tablet f70366bad94f40ff82e890aa5a0bb769
I20260929 16:20:59.544221 12175 tablet_service.cc:1514] Processing DeleteTablet for tablet f70366bad94f40ff82e890aa5a0bb769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:59 UTC) from {username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:48942
I20260929 16:20:59.544468 11900 tablet_service.cc:1514] Processing DeleteTablet for tablet f70366bad94f40ff82e890aa5a0bb769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:59 UTC) from {username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:39772
I20260929 16:20:59.544670 12360 tablet_replica.cc:330] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: stopping tablet replica
I20260929 16:20:59.544682 12037 tablet_service.cc:1514] Processing DeleteTablet for tablet f70366bad94f40ff82e890aa5a0bb769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:20:59 UTC) from {username='oryx', principal='oryx/127.4.200.124@KRBTEST.COM'} at 127.0.0.1:41604
I20260929 16:20:59.544886 12360 raft_consensus.cc:2229] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:59.545014 12361 tablet_replica.cc:330] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a: stopping tablet replica
I20260929 16:20:59.545132 12360 raft_consensus.cc:2258] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:59.545153 12361 raft_consensus.cc:2229] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:20:59.545274 12361 raft_consensus.cc:2258] T f70366bad94f40ff82e890aa5a0bb769 P dc685fd96de045cf8557c575cf51477a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:59.545506 12362 tablet_replica.cc:330] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: stopping tablet replica
I20260929 16:20:59.545598 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11748
I20260929 16:20:59.545627 12362 raft_consensus.cc:2229] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:20:59.546485 12362 raft_consensus.cc:2258] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:20:59.547142 12360 ts_tablet_manager.cc:1905] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:59.547358 12362 ts_tablet_manager.cc:1905] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:20:59.549685 12362 ts_tablet_manager.cc:1918] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:59.549685 12360 ts_tablet_manager.cc:1918] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:20:59.549765 12362 log.cc:1196] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/f70366bad94f40ff82e890aa5a0bb769
I20260929 16:20:59.549765 12360 log.cc:1196] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/f70366bad94f40ff82e890aa5a0bb769
I20260929 16:20:59.550076 12360 ts_tablet_manager.cc:1939] T f70366bad94f40ff82e890aa5a0bb769 P 5f3a71ad6ca142f3862c9e493424ffd2: Deleting consensus metadata
I20260929 16:20:59.550087 12362 ts_tablet_manager.cc:1939] T f70366bad94f40ff82e890aa5a0bb769 P 5218701608d04deca1175445803d1a10: Deleting consensus metadata
I20260929 16:20:59.551047 11771 catalog_manager.cc:4841] TS 5218701608d04deca1175445803d1a10 (127.4.200.66:43115): tablet f70366bad94f40ff82e890aa5a0bb769 (table test-table [id=806157163a9449d8aa32a5c6e4b51fde]) successfully deleted
I20260929 16:20:59.551261 11774 catalog_manager.cc:4841] TS 5f3a71ad6ca142f3862c9e493424ffd2 (127.4.200.67:39947): tablet f70366bad94f40ff82e890aa5a0bb769 (table test-table [id=806157163a9449d8aa32a5c6e4b51fde]) successfully deleted
I20260929 16:20:59.553925 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11974
I20260929 16:20:59.554044 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:20:59.553752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44163 (local address 127.0.0.1:39786)
0929 16:20:59.553881 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:20:59.553941 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:20:59.555497 11771 catalog_manager.cc:4584] TS dc685fd96de045cf8557c575cf51477a (127.4.200.65:44163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f70366bad94f40ff82e890aa5a0bb769: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44163: connect: Connection refused (error 111)
I20260929 16:20:59.559243 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 12110
I20260929 16:20:59.565795 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11465
I20260929 16:20:59.571602 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 11570
2026-09-29T16:20:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5686 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:20:59 dist-test-slave-s9ct krb5kdc[12365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:20:59 dist-test-slave-s9ct krb5kdc[12365](info): set up 2 sockets
Sep 29 16:20:59 dist-test-slave-s9ct krb5kdc[12365](info): commencing operation
krb5kdc: starting...
W20260929 16:21:00.615039 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:21:00 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab.
Entry for principal oryx/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab.
I20260929 16:21:00.738112 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43895
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab
--principal=oryx/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:43895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:00.848809 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:00.849143 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:00.849234 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:00.853368 12378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:00.853468 12378 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:21:00.853497 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:00.853541 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:00.853575 12378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:00.853606 12378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:00.859123 12378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:43895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.126.keytab
--principal=oryx/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:00.860252 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:00.861299 12378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:00.870220 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:00.870232 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:00.870316 12378 server_base.cc:1034] running on GCE node
W20260929 16:21:00.870246 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:00.871165 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:00.872279 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:00.873479 12378 hybrid_clock.cc:648] HybridClock initialized: now 1790698860873459 us; error 43 us; skew 500 ppm
Sep 29 16:21:00 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:00.876235 12378 init.cc:377] Logged in from keytab as oryx/127.4.200.126@KRBTEST.COM (short username oryx)
I20260929 16:21:00.877190 12378 webserver.cc:460] Webserver started at http://127.4.200.126:41467/ using document root <none> and password file <none>
I20260929 16:21:00.877767 12378 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:00.877859 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:00.878100 12378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:00.880231 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "4f88e6058e13488d97045106e9debc58"
format_stamp: "Formatted at 2026-09-29 16:21:00 on dist-test-slave-s9ct"
server_key: "b458821d2e3faa899a1df79d519db9c7"
server_key_iv: "057465d63354479f85e04c8e6277c812"
server_key_version: "encryptionkey@0"
I20260929 16:21:00.880815 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "4f88e6058e13488d97045106e9debc58"
format_stamp: "Formatted at 2026-09-29 16:21:00 on dist-test-slave-s9ct"
server_key: "b458821d2e3faa899a1df79d519db9c7"
server_key_iv: "057465d63354479f85e04c8e6277c812"
server_key_version: "encryptionkey@0"
I20260929 16:21:00.884712 12378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:21:00.887254 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:00.888489 12378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:00.888639 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "4f88e6058e13488d97045106e9debc58"
format_stamp: "Formatted at 2026-09-29 16:21:00 on dist-test-slave-s9ct"
server_key: "b458821d2e3faa899a1df79d519db9c7"
server_key_iv: "057465d63354479f85e04c8e6277c812"
server_key_version: "encryptionkey@0"
I20260929 16:21:00.888754 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:00.905951 12378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:00.908773 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:00.908979 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:00.917054 12378 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:43895
I20260929 16:21:00.917090 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:43895 every 8 connection(s)
I20260929 16:21:00.918018 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:00.920789 12456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:00.924486 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 12378
I20260929 16:21:00.924594 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:21:00.924965 4897 external_mini_cluster.cc:1428] Setting key 9e72a837041580a3b037ddb77bb793ed
I20260929 16:21:00.926832 12456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Bootstrap starting.
I20260929 16:21:00.929090 12456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:00.930070 12456 log.cc:824] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:00.932451 12456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: No bootstrap required, opened a new log
Sep 29 16:21:00 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:00.937114 12456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } }
I20260929 16:21:00.937427 12456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:00.937489 12456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f88e6058e13488d97045106e9debc58, State: Initialized, Role: FOLLOWER
I20260929 16:21:00.938040 12456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } }
I20260929 16:21:00.938180 12456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:00.938252 12456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:00.938355 12456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:00.939631 12456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } }
I20260929 16:21:00.940029 12456 leader_election.cc:302] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f88e6058e13488d97045106e9debc58; no voters:
I20260929 16:21:00.940367 12456 leader_election.cc:288] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:21:00.940488 12461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:00.940732 12461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [term 1 LEADER]: Becoming Leader. State: Replica: 4f88e6058e13488d97045106e9debc58, State: Running, Role: LEADER
I20260929 16:21:00.941049 12461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } }
I20260929 16:21:00.941697 12456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:00.941776 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:00.926546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.4.200.126:43895)
0929 16:21:00.927057 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:00.927074 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:00.927113 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:21:00.927776 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:00.927793 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:00.928135 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:00.928489 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:00.929364 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:00.930837 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:00.931748 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:00.932453 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:00.932637 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:00.935614 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:00.935623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:00.935643 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:00.935680 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:00.937816 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:00.938385 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:00.938421 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:00.938428 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:00.938589 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:00.938877 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:00.938880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:00.938882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:00.939435 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:00.939651 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:00.940328 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 16:21:00.940727 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260929 16:21:00.942342 12461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f88e6058e13488d97045106e9debc58. Latest consensus state: current_term: 1 leader_uuid: "4f88e6058e13488d97045106e9debc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } } }
I20260929 16:21:00.942490 12461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:00.944265 12462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f88e6058e13488d97045106e9debc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f88e6058e13488d97045106e9debc58" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 43895 } } }
I20260929 16:21:00.944367 12462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:00.944892 12465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:00.947619 12465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:00.953389 12465 catalog_manager.cc:1347] Generated new cluster ID: e327fd3f7e0a40929680bc262f9a206d
I20260929 16:21:00.953476 12465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:00.995787 12465 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:21:00.996842 12465 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:01.010186 12465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Generated new TSK 0
I20260929 16:21:01.011039 12465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
I20260929 16:21:01.041769 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43895
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:01.172065 12482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:01.172302 12482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:01.172346 12482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:01.176291 12482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:01.176339 12482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:01.176376 12482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:21:01.176426 12482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:01.181420 12482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:43895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:01.182363 12482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:01.183411 12482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:01.193504 12488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:01.193559 12490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:01.193513 12487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:01.193787 12482 server_base.cc:1034] running on GCE node
I20260929 16:21:01.194293 12482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:01.194914 12482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:01.196079 12482 hybrid_clock.cc:648] HybridClock initialized: now 1790698861196039 us; error 51 us; skew 500 ppm
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:01.198943 12482 init.cc:377] Logged in from keytab as oryx/127.4.200.65@KRBTEST.COM (short username oryx)
I20260929 16:21:01.199785 12482 webserver.cc:460] Webserver started at http://127.4.200.65:41655/ using document root <none> and password file <none>
I20260929 16:21:01.200307 12482 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:01.200369 12482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:01.200553 12482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:01.202618 12482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "525990dd76f8428bba9963cf78a89d2d"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "6005da268d4efb3fa288e30531e0a434"
server_key_iv: "3c91869e992e3005a681eff696112cd9"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.203135 12482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "525990dd76f8428bba9963cf78a89d2d"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "6005da268d4efb3fa288e30531e0a434"
server_key_iv: "3c91869e992e3005a681eff696112cd9"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.206950 12482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:21:01.209256 12497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.210647 12482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:01.210759 12482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "525990dd76f8428bba9963cf78a89d2d"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "6005da268d4efb3fa288e30531e0a434"
server_key_iv: "3c91869e992e3005a681eff696112cd9"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.210851 12482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:01.236214 12482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:01.239863 12482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:01.240032 12482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:01.241593 12482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:01.241676 12482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.241751 12482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:01.241808 12482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:01.254112 12482 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:41697
I20260929 16:21:01.254151 12612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:41697 every 8 connection(s)
I20260929 16:21:01.255283 12482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:01.256630 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.242273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:44173 (local address 127.4.200.126:43895)
0929 16:21:01.242463 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.242470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.243324 (+ 854us) server_negotiation.cc:407] Connection header received
0929 16:21:01.244272 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.244276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.244335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.244451 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.245682 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.246619 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.247854 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.248272 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.248412 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.251719 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.251725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.251730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.251771 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.253744 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.254491 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.254495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.254503 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.254601 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.255059 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.255061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.255063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.255273 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.255408 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.256354 (+ 946us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.256508 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:21:01.257156 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.242701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.65:44173)
0929 16:21:01.243166 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.243213 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.244046 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.244618 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.244624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.245012 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.245540 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.245553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.246770 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.246773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.247743 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.247753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.248445 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.248448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.248586 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.249235 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.249254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.251530 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.253920 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.253940 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.253956 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.254315 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.254723 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.254726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.254728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.254903 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.255400 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.255407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.255730 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.255957 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.256291 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260929 16:21:01.258504 12613 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43895
I20260929 16:21:01.258661 12613 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:01.259218 12613 heartbeater.cc:511] Master 127.4.200.126:43895 requested a full tablet report, sending...
I20260929 16:21:01.259443 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 12482
I20260929 16:21:01.259570 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:21:01.259920 4897 external_mini_cluster.cc:1428] Setting key 4a2ff00ca764d11588a2c92f1bca8e1e
I20260929 16:21:01.260991 12409 ts_manager.cc:194] Registered new tserver with Master: 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697)
I20260929 16:21:01.262591 12409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:44173
I20260929 16:21:01.268636 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.260733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:55453 (local address 127.4.200.126:43895)
0929 16:21:01.261165 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.261169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.261182 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:21:01.261299 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.261303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.261343 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.261437 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.261865 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.262439 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.263337 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.263638 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.263742 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.265269 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.265272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.265274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.265295 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.266960 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.267484 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.267487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.267488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.267560 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.267872 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.267874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.267876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.268032 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.268118 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.268417 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.268514 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":368}
I20260929 16:21:01.268766 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.260761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.65:55453)
0929 16:21:01.261067 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.261083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.261172 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.261452 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.261455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.261657 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.261754 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.261759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.262554 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.262557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.263216 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.263225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.263774 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.263779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.263949 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.264603 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.264624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.265126 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.267082 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.267094 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.267099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.267382 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.267653 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.267659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.267664 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.267773 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.268133 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.268139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.268253 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.268372 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.268533 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
I20260929 16:21:01.290685 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43895
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:01.403920 12622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:01.404232 12622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:01.404316 12622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:01.408373 12622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:01.408455 12622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:01.408553 12622 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:21:01.408629 12622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:01.414012 12622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:43895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:01.415091 12622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:01.416086 12622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:01.425524 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:01.425556 12622 server_base.cc:1034] running on GCE node
W20260929 16:21:01.425740 12627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:01.425907 12628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:01.426298 12622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:01.426924 12622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:01.428105 12622 hybrid_clock.cc:648] HybridClock initialized: now 1790698861428052 us; error 49 us; skew 500 ppm
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:01.430883 12622 init.cc:377] Logged in from keytab as oryx/127.4.200.66@KRBTEST.COM (short username oryx)
I20260929 16:21:01.431666 12622 webserver.cc:460] Webserver started at http://127.4.200.66:36491/ using document root <none> and password file <none>
I20260929 16:21:01.432170 12622 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:01.432228 12622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:01.432381 12622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:01.434281 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "cdcab6fbd2034ef69033112033a2a3ee"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "09544cd882b498d79aefc9efd3c812d8"
server_key_iv: "276e0e7295e5ec77bc755e47664be1c2"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.434847 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "cdcab6fbd2034ef69033112033a2a3ee"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "09544cd882b498d79aefc9efd3c812d8"
server_key_iv: "276e0e7295e5ec77bc755e47664be1c2"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.438743 12622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:21:01.441068 12637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.442456 12622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:21:01.442574 12622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "cdcab6fbd2034ef69033112033a2a3ee"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "09544cd882b498d79aefc9efd3c812d8"
server_key_iv: "276e0e7295e5ec77bc755e47664be1c2"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.442685 12622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:01.466106 12622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:01.469322 12622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:01.469460 12622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:01.470856 12622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:01.470934 12622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.471022 12622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:01.471072 12622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:01.481848 12622 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:37325
I20260929 16:21:01.482081 12752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:37325 every 8 connection(s)
I20260929 16:21:01.482949 12622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:01.484551 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.471515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:36717 (local address 127.4.200.126:43895)
0929 16:21:01.471669 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.471673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.472583 (+ 910us) server_negotiation.cc:407] Connection header received
0929 16:21:01.473894 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.473898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.473965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.474090 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.475249 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.475818 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.477145 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.477432 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.477543 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.480831 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.480834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.480837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.480872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.482516 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.482988 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.482993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.482995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.483085 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.483441 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.483446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.483448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.483702 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.483805 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.484326 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.484426 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:21:01.485277 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.471939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.66:36717)
0929 16:21:01.472410 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.472464 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.473616 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.474261 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.474267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.474655 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.475106 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.475119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.475949 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.475952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.477015 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.477025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.477575 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.477579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.477768 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.478320 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.478340 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.480667 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.482637 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.482642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.482653 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.482883 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.483189 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.483192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.483194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.483337 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.483796 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.483802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.484083 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.484253 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.484482 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260929 16:21:01.486470 12753 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43895
I20260929 16:21:01.486625 12753 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:01.486955 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 12622
I20260929 16:21:01.487082 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:21:01.487197 12753 heartbeater.cc:511] Master 127.4.200.126:43895 requested a full tablet report, sending...
I20260929 16:21:01.487469 4897 external_mini_cluster.cc:1428] Setting key 237e66f2a89eb2fdb0c5e3c5f9e238f2
I20260929 16:21:01.488310 12410 ts_manager.cc:194] Registered new tserver with Master: cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325)
I20260929 16:21:01.489117 12410 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:36717
I20260929 16:21:01.496454 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.488771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:44997 (local address 127.4.200.126:43895)
0929 16:21:01.488919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.488923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.489077 (+ 154us) server_negotiation.cc:407] Connection header received
0929 16:21:01.489171 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.489176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.489219 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.489299 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.490005 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.490752 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.491938 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.492152 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.492265 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.493322 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.493326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.493328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.493350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.494817 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.495250 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.495255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.495257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.495339 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.495632 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.495636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.495637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.495841 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.495954 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.496208 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.496309 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:21:01.496469 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.488724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.66:44997)
0929 16:21:01.488980 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.488997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.489093 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.489455 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.489461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.489690 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.489874 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.489885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.490871 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.490876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.491827 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.491839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.492270 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.492274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.492363 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.492857 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.492868 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.493194 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.494946 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.494949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.494952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.495145 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.495442 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.495445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.495447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.495525 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.495942 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.495945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.496035 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.496158 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.496298 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
I20260929 16:21:01.517436 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:43895
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:01.628887 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:01.629123 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:01.629202 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:01.633163 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:01.633209 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:01.633247 12762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:21:01.633296 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:01.638322 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.200.126:43895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:01.639326 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:01.640316 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:21:01.649999 12762 server_base.cc:1034] running on GCE node
W20260929 16:21:01.649952 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:01.649961 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:01.649938 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:01.650538 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:01.651062 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:01.652225 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790698861652196 us; error 39 us; skew 500 ppm
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:01.655078 12762 init.cc:377] Logged in from keytab as oryx/127.4.200.67@KRBTEST.COM (short username oryx)
I20260929 16:21:01.655913 12762 webserver.cc:460] Webserver started at http://127.4.200.67:37263/ using document root <none> and password file <none>
I20260929 16:21:01.656423 12762 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:01.656490 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:01.656656 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:01.658782 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "d49b5ab2815448729d641f29a0abfc62"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "cef1aaf89632615239c89c40a27f43ef"
server_key_iv: "4673c4f3b72280e01943fff8aede199e"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.659369 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "d49b5ab2815448729d641f29a0abfc62"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "cef1aaf89632615239c89c40a27f43ef"
server_key_iv: "4673c4f3b72280e01943fff8aede199e"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.663079 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:21:01.665517 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.666842 12762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:01.666981 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "d49b5ab2815448729d641f29a0abfc62"
format_stamp: "Formatted at 2026-09-29 16:21:01 on dist-test-slave-s9ct"
server_key: "cef1aaf89632615239c89c40a27f43ef"
server_key_iv: "4673c4f3b72280e01943fff8aede199e"
server_key_version: "encryptionkey@0"
I20260929 16:21:01.667085 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:01.680464 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:01.683480 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:01.683688 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:01.685194 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:01.685246 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:01.685319 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:01.685356 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:01.697733 12762 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:44223
I20260929 16:21:01.697768 12892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:44223 every 8 connection(s)
I20260929 16:21:01.698756 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:01.699508 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.686049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:42219 (local address 127.4.200.126:43895)
0929 16:21:01.686235 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.686243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.687156 (+ 913us) server_negotiation.cc:407] Connection header received
0929 16:21:01.688476 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.688480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.688534 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.688629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.689775 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.690493 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.691614 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.691919 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.692031 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.695420 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.695424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.695427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.695454 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.697409 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.697976 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.697980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.697982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.698069 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.698443 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.698446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.698448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.698636 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.698743 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.699219 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.699339 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:21:01.700376 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.686485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.67:42219)
0929 16:21:01.686965 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.687025 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.688221 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.688842 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.688851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.689180 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.689626 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.689639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.690627 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.690630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.691486 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.691495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.692066 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.692071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.692302 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.692973 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.692994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.695252 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.697547 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.697553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.697567 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.697869 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.698176 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.698179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.698181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.698311 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.698749 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.698758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.698994 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.699175 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.699504 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260929 16:21:01.701534 12893 heartbeater.cc:347] Connected to a master server at 127.4.200.126:43895
I20260929 16:21:01.701709 12893 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:01.702205 12893 heartbeater.cc:511] Master 127.4.200.126:43895 requested a full tablet report, sending...
I20260929 16:21:01.703305 12409 ts_manager.cc:194] Registered new tserver with Master: d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:01.703740 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 12762
I20260929 16:21:01.703876 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:21:01.704172 12409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:42219
I20260929 16:21:01.704290 4897 external_mini_cluster.cc:1428] Setting key e4db80d2bc184b7813e2b66a885569c5
I20260929 16:21:01.706125 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:21:01.711580 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.703667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:37821 (local address 127.4.200.126:43895)
0929 16:21:01.703777 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.703781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.703978 (+ 197us) server_negotiation.cc:407] Connection header received
0929 16:21:01.704169 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.704172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.704238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.704331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.705003 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.705724 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.706793 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.707184 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.707306 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.708498 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.708501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.708503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.708523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.709870 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.710346 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.710348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.710350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.710466 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.710775 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.710778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.710779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.710976 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.711079 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.711380 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.711470 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:21:01.711711 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.703587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.4.200.67:37821)
0929 16:21:01.703899 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.703915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.704047 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.704447 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.704450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.704701 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:01.704835 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.704844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.705899 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.705902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.706678 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.706688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.707306 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.707309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.707382 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.707890 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:01.707902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:01.708359 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:01.709975 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.709981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.709986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.710249 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.710561 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:01.710567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:01.710572 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:01.710684 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:01.711088 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:01.711093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:01.711210 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.711333 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.711489 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:21:01.730886 12459 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:01.724929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.4.200.126:43895)
0929 16:21:01.725063 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.725066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.725078 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:21:01.725117 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.725119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.725153 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.725243 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.725933 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.726502 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.727330 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.727546 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.727659 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.730777 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48572: BlockingRecv error: recv got EOF from 127.0.0.1:48572 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 16:21:01 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:01.739866 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.732322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.4.200.126:43895)
0929 16:21:01.732450 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.732454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.732612 (+ 158us) server_negotiation.cc:407] Connection header received
0929 16:21:01.732772 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.732775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.732819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.732897 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:01.733465 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.733967 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.734810 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.735004 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.735075 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.737137 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:01.737140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:01.737142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:01.737162 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:01.738590 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.738948 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.738951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.738952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.739036 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:01.739250 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:01.739252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:01.739254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:01.739408 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:01.739498 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.739683 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.739767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:21:01.742475 12409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:21:01.745646 12409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:01.760131 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.754803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.4.200.67:44223)
0929 16:21:01.755152 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.755165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.755370 (+ 205us) server_negotiation.cc:407] Connection header received
0929 16:21:01.755527 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.755533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.755703 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.755871 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:01.756148 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.756963 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.758496 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.759302 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.759426 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.759567 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.759890 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.759984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260929 16:21:01.760130 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.754638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:44223 (local address 127.0.0.1:53082)
0929 16:21:01.755278 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.755288 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.755397 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.755869 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.755874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.755896 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:01.756021 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.756028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.757072 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.757077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.758338 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.758348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.759620 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.759623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.759705 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.759733 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.759857 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.759974 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260929 16:21:01.760515 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.754019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.0.0.1:45888)
0929 16:21:01.754698 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.754711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.754798 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.756194 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.756197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.756217 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:01.756296 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.756301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.757716 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.757722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.759052 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.759063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.760112 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.760116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.760220 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.760252 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.760326 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.760389 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":86,"thread_start_us":99,"threads_started":1}
I20260929 16:21:01.761108 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.753907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:41697 (local address 127.0.0.1:47590)
0929 16:21:01.754223 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.754275 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.754420 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.756013 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.756017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.756034 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:01.756138 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.756147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.757722 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.757725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.758853 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.758862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.760115 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.760118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.760828 (+ 710us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.760850 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.760917 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.760977 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":39,"thread_start_us":63,"threads_started":1}
I20260929 16:21:01.761471 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.754295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.4.200.65:41697)
0929 16:21:01.755660 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.755673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.755689 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:21:01.755747 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.755751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.755879 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.756024 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:01.756268 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.757454 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.758968 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.759804 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.759914 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.760058 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.761260 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.761354 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":107,"threads_started":1}
I20260929 16:21:01.760979 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.754693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.4.200.66:37325)
0929 16:21:01.755052 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.755065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.755088 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:21:01.755183 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.755188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.755396 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.756223 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:01.756482 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.757555 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.759180 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.759978 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.760164 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.760675 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.760730 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.760819 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":82,"threads_started":1}
I20260929 16:21:01.763216 12815 tablet_service.cc:1467] Processing CreateTablet for tablet e568635903234c67bc078096f8a58120 (DEFAULT_TABLE table=test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.764353 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e568635903234c67bc078096f8a58120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.764117 12547 tablet_service.cc:1467] Processing CreateTablet for tablet e568635903234c67bc078096f8a58120 (DEFAULT_TABLE table=test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.764511 12687 tablet_service.cc:1467] Processing CreateTablet for tablet e568635903234c67bc078096f8a58120 (DEFAULT_TABLE table=test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.765049 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e568635903234c67bc078096f8a58120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.765408 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e568635903234c67bc078096f8a58120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.772079 12922 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Bootstrap starting.
I20260929 16:21:01.772249 12921 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Bootstrap starting.
I20260929 16:21:01.773069 12923 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Bootstrap starting.
I20260929 16:21:01.774060 12921 tablet_bootstrap.cc:654] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.774785 12923 tablet_bootstrap.cc:654] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.775151 12921 log.cc:824] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:01.775586 12923 log.cc:824] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:01.776288 12922 tablet_bootstrap.cc:654] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.776925 12922 log.cc:824] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:01.777529 12923 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: No bootstrap required, opened a new log
I20260929 16:21:01.777724 12923 ts_tablet_manager.cc:1397] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:21:01.778594 12921 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: No bootstrap required, opened a new log
I20260929 16:21:01.778779 12922 tablet_bootstrap.cc:492] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: No bootstrap required, opened a new log
I20260929 16:21:01.778813 12921 ts_tablet_manager.cc:1397] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:21:01.778950 12922 ts_tablet_manager.cc:1397] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:21:01.781672 12923 raft_consensus.cc:348] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.781929 12923 raft_consensus.cc:374] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.781986 12923 raft_consensus.cc:729] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.781790 12921 raft_consensus.cc:348] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.782485 12923 consensus_queue.cc:260] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.782635 12921 raft_consensus.cc:374] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.782709 12921 raft_consensus.cc:729] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.782625 12922 raft_consensus.cc:348] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.782866 12922 raft_consensus.cc:374] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.782922 12922 raft_consensus.cc:729] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.783190 12921 consensus_queue.cc:260] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.783299 12922 consensus_queue.cc:260] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.783485 12923 ts_tablet_manager.cc:1428] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 16:21:01.783831 12613 heartbeater.cc:503] Master 127.4.200.126:43895 was elected leader, sending a full tablet report...
I20260929 16:21:01.783831 12753 heartbeater.cc:503] Master 127.4.200.126:43895 was elected leader, sending a full tablet report...
I20260929 16:21:01.784121 12922 ts_tablet_manager.cc:1428] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:21:01.784410 12893 heartbeater.cc:503] Master 127.4.200.126:43895 was elected leader, sending a full tablet report...
I20260929 16:21:01.785079 12921 ts_tablet_manager.cc:1428] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:21:01.842653 12928 raft_consensus.cc:482] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:01.842923 12928 raft_consensus.cc:504] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.844267 12928 leader_election.cc:288] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:01.849071 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.844485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.4.200.65:40609)
0929 16:21:01.844934 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.844946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.845081 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.845366 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.845369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.845384 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:01.845559 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.845564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.846423 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.846430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.847888 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.847899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.848578 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.848580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.848696 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.848725 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.848805 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.848891 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":92,"spinlock_wait_cycles":4480,"thread_start_us":170,"threads_started":1}
I20260929 16:21:01.849298 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.844631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40609 (local address 127.4.200.66:37325)
0929 16:21:01.844850 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.844855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.845029 (+ 174us) server_negotiation.cc:407] Connection header received
0929 16:21:01.845217 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.845225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.845267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.845354 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:01.845667 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.846288 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.848015 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.848454 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.848582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.848651 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.849044 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.849135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 16:21:01.849936 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.844678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:45465 (local address 127.4.200.67:44223)
0929 16:21:01.844827 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.844832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.845064 (+ 232us) server_negotiation.cc:407] Connection header received
0929 16:21:01.845230 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.845235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.845281 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.845364 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:01.845664 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.846444 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.848680 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.849214 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.849340 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.849393 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.849741 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.849823 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:21:01.849952 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.844532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:44223 (local address 127.4.200.65:45465)
0929 16:21:01.844964 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:01.844975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:01.845104 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:01.845374 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:01.845377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:01.845391 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:01.845551 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.845557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.846555 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.846561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.847893 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:01.847899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.849457 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.849460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:01.849590 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.849620 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:21:01.849701 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:21:01.849772 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":55,"thread_start_us":130,"threads_started":1}
I20260929 16:21:01.850188 12707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e568635903234c67bc078096f8a58120" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee" is_pre_election: true
I20260929 16:21:01.850579 12707 raft_consensus.cc:2454] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 0.
I20260929 16:21:01.850577 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e568635903234c67bc078096f8a58120" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62" is_pre_election: true
I20260929 16:21:01.850850 12846 raft_consensus.cc:2454] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 0.
I20260929 16:21:01.851089 12501 leader_election.cc:302] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:01.851384 12928 raft_consensus.cc:2781] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:01.851466 12928 raft_consensus.cc:482] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:01.851497 12928 raft_consensus.cc:3037] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:01.852543 12928 raft_consensus.cc:504] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.852918 12928 leader_election.cc:288] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 election: Requested vote from peers cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:01.853339 12707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e568635903234c67bc078096f8a58120" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee"
I20260929 16:21:01.853339 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e568635903234c67bc078096f8a58120" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62"
I20260929 16:21:01.853468 12846 raft_consensus.cc:3037] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:01.853498 12707 raft_consensus.cc:3037] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:01.854893 12846 raft_consensus.cc:2454] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 1.
I20260929 16:21:01.855250 12498 leader_election.cc:302] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, d49b5ab2815448729d641f29a0abfc62; no voters:
I20260929 16:21:01.855347 12707 raft_consensus.cc:2454] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 1.
I20260929 16:21:01.855459 12928 raft_consensus.cc:2781] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:01.855755 12928 raft_consensus.cc:686] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 1 LEADER]: Becoming Leader. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Running, Role: LEADER
I20260929 16:21:01.856292 12928 consensus_queue.cc:237] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.859705 12410 catalog_manager.cc:5495] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 525990dd76f8428bba9963cf78a89d2d (127.4.200.65). New cstate: current_term: 1 leader_uuid: "525990dd76f8428bba9963cf78a89d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } health_report { overall_health: UNKNOWN } } }
I20260929 16:21:01.881677 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:01.878186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.4.200.65:41697)
0929 16:21:01.878321 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:01.878326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:01.878340 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:21:01.878465 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:01.878470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:01.878532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:01.878661 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:01.878903 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.879408 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.880255 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:01.880477 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:01.880583 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:01.880869 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:01.881284 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:01.881409 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:01.881467 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:21:01.881520 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:21:01.887373 12707 raft_consensus.cc:1261] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Refusing update from remote peer 525990dd76f8428bba9963cf78a89d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:01.887691 12846 raft_consensus.cc:1261] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Refusing update from remote peer 525990dd76f8428bba9963cf78a89d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:01.888075 12932 consensus_queue.cc:1035] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:01.888250 12928 consensus_queue.cc:1035] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:01.897239 12936 mvcc.cc:205] Tried to move back new op lower bound from 7334702538283237376 to 7334702538166128640. Current Snapshot: MvccSnapshot[applied={T|T < 7334702538283237376}]
I20260929 16:21:01.904955 12409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:01.905144 12409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:01.907584 12409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Sending DeleteTablet for 3 replicas of tablet e568635903234c67bc078096f8a58120
I20260929 16:21:01.908418 12687 tablet_service.cc:1514] Processing DeleteTablet for tablet e568635903234c67bc078096f8a58120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:01 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:45888
I20260929 16:21:01.908447 12547 tablet_service.cc:1514] Processing DeleteTablet for tablet e568635903234c67bc078096f8a58120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:01 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:47590
I20260929 16:21:01.908456 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet e568635903234c67bc078096f8a58120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:01 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260929 16:21:01.909701 12409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:21:01.910101 12409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:01.910640 12945 tablet_replica.cc:330] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: stopping tablet replica
I20260929 16:21:01.910656 12946 tablet_replica.cc:330] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: stopping tablet replica
I20260929 16:21:01.910977 12945 raft_consensus.cc:2229] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:21:01.911150 12946 raft_consensus.cc:2229] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:01.911410 12945 raft_consensus.cc:2258] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:01.912533 12946 raft_consensus.cc:2258] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:01.912999 12947 tablet_replica.cc:330] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: stopping tablet replica
I20260929 16:21:01.913347 12947 raft_consensus.cc:2229] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:01.913681 12947 raft_consensus.cc:2258] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:01.913969 12945 ts_tablet_manager.cc:1905] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:01.915138 12947 ts_tablet_manager.cc:1905] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:01.916237 12687 tablet_service.cc:1467] Processing CreateTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (DEFAULT_TABLE table=test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.916512 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b3606ffdf4d4e8d1b25baf0e1f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.917052 12945 ts_tablet_manager.cc:1918] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:01.917164 12945 log.cc:1196] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/e568635903234c67bc078096f8a58120
I20260929 16:21:01.917464 12945 ts_tablet_manager.cc:1939] T e568635903234c67bc078096f8a58120 P 525990dd76f8428bba9963cf78a89d2d: Deleting consensus metadata
I20260929 16:21:01.918128 12947 ts_tablet_manager.cc:1918] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:01.918272 12947 log.cc:1196] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/e568635903234c67bc078096f8a58120
I20260929 16:21:01.918670 12947 ts_tablet_manager.cc:1939] T e568635903234c67bc078096f8a58120 P cdcab6fbd2034ef69033112033a2a3ee: Deleting consensus metadata
I20260929 16:21:01.919093 12923 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Bootstrap starting.
I20260929 16:21:01.919950 12923 tablet_bootstrap.cc:654] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.920769 12397 catalog_manager.cc:4841] TS cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325): tablet e568635903234c67bc078096f8a58120 (table test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]) successfully deleted
I20260929 16:21:01.921068 12923 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: No bootstrap required, opened a new log
I20260929 16:21:01.921145 12923 ts_tablet_manager.cc:1397] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:21:01.921190 12394 catalog_manager.cc:4841] TS 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697): tablet e568635903234c67bc078096f8a58120 (table test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]) successfully deleted
I20260929 16:21:01.921294 12815 tablet_service.cc:1467] Processing CreateTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (DEFAULT_TABLE table=test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.921540 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b3606ffdf4d4e8d1b25baf0e1f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.921494 12547 tablet_service.cc:1467] Processing CreateTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (DEFAULT_TABLE table=test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:01.921521 12923 raft_consensus.cc:348] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.921624 12923 raft_consensus.cc:374] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.921656 12923 raft_consensus.cc:729] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.921729 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42b3606ffdf4d4e8d1b25baf0e1f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:01.921770 12923 consensus_queue.cc:260] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.921983 12923 ts_tablet_manager.cc:1428] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:21:01.923830 12921 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Bootstrap starting.
I20260929 16:21:01.924767 12921 tablet_bootstrap.cc:654] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.925318 12922 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Bootstrap starting.
I20260929 16:21:01.925875 12946 ts_tablet_manager.cc:1905] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:01.926165 12922 tablet_bootstrap.cc:654] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:01.927310 12922 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: No bootstrap required, opened a new log
I20260929 16:21:01.927385 12922 ts_tablet_manager.cc:1397] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:21:01.927790 12922 raft_consensus.cc:348] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.927892 12922 raft_consensus.cc:374] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.927922 12922 raft_consensus.cc:729] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.928027 12922 consensus_queue.cc:260] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.928188 12946 ts_tablet_manager.cc:1918] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:01.928259 12922 ts_tablet_manager.cc:1428] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:01.928310 12946 log.cc:1196] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/e568635903234c67bc078096f8a58120
I20260929 16:21:01.928573 12946 ts_tablet_manager.cc:1939] T e568635903234c67bc078096f8a58120 P d49b5ab2815448729d641f29a0abfc62: Deleting consensus metadata
I20260929 16:21:01.929725 12394 catalog_manager.cc:4841] TS d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223): tablet e568635903234c67bc078096f8a58120 (table test-table [id=f9dc980d4e7440ad9f567e1dc233ac23]) successfully deleted
I20260929 16:21:01.935307 12921 tablet_bootstrap.cc:492] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: No bootstrap required, opened a new log
I20260929 16:21:01.935402 12921 ts_tablet_manager.cc:1397] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260929 16:21:01.935873 12921 raft_consensus.cc:348] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.935990 12921 raft_consensus.cc:374] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:01.936021 12921 raft_consensus.cc:729] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Initialized, Role: FOLLOWER
I20260929 16:21:01.936206 12921 consensus_queue.cc:260] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:01.936507 12921 ts_tablet_manager.cc:1428] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260929 16:21:01.950023 12894 tablet.cc:2369] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:21:01.984432 12754 tablet.cc:2369] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:21:02.006649 12614 tablet.cc:2369] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:21:02.108122 12927 raft_consensus.cc:482] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:02.108282 12927 raft_consensus.cc:504] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.109287 12927 leader_election.cc:288] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.113655 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.109705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:51205 (local address 127.4.200.65:41697)
0929 16:21:02.109836 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.109839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.110021 (+ 182us) server_negotiation.cc:407] Connection header received
0929 16:21:02.110087 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.110091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.110163 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.110261 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:02.110567 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.111392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.112657 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.113087 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.113185 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.113262 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.113449 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.113526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:21:02.113688 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.110074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:43923 (local address 127.4.200.67:44223)
0929 16:21:02.110217 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.110222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.110237 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:02.110428 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.110431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.110479 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.110569 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:02.110791 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.111482 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.112743 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.113125 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.113259 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.113310 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.113514 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.113586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:21:02.113672 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.109581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:41697 (local address 127.4.200.66:51205)
0929 16:21:02.109837 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.109854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.109971 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.110288 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.110290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.110304 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:02.110463 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.110468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.111505 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.111509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.112537 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.112546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.113212 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.113219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.113310 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.113341 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.113410 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.113496 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260929 16:21:02.113694 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.109955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:44223 (local address 127.4.200.66:43923)
0929 16:21:02.110173 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.110188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.110289 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.110604 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.110606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.110621 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:02.110693 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.110697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.111621 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.111624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.112582 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.112619 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.113267 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.113271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.113366 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.113383 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.113478 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.113540 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260929 16:21:02.114420 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b3606ffdf4d4e8d1b25baf0e1f682" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d" is_pre_election: true
I20260929 16:21:02.114466 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b3606ffdf4d4e8d1b25baf0e1f682" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62" is_pre_election: true
I20260929 16:21:02.114614 12846 raft_consensus.cc:2454] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 0.
I20260929 16:21:02.114675 12565 raft_consensus.cc:2454] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 0.
I20260929 16:21:02.114984 12638 leader_election.cc:302] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdcab6fbd2034ef69033112033a2a3ee, d49b5ab2815448729d641f29a0abfc62; no voters:
I20260929 16:21:02.115247 12927 raft_consensus.cc:2781] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:02.115310 12927 raft_consensus.cc:482] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:02.115348 12927 raft_consensus.cc:3037] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.116354 12927 raft_consensus.cc:504] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.116684 12927 leader_election.cc:288] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 election: Requested vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.117038 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b3606ffdf4d4e8d1b25baf0e1f682" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62"
I20260929 16:21:02.117038 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42b3606ffdf4d4e8d1b25baf0e1f682" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d"
I20260929 16:21:02.117154 12565 raft_consensus.cc:3037] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.117161 12846 raft_consensus.cc:3037] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.118230 12565 raft_consensus.cc:2454] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 1.
I20260929 16:21:02.118230 12846 raft_consensus.cc:2454] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 1.
I20260929 16:21:02.118496 12638 leader_election.cc:302] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:02.118680 12927 raft_consensus.cc:2781] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:02.118846 12927 raft_consensus.cc:686] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 1 LEADER]: Becoming Leader. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Running, Role: LEADER
I20260929 16:21:02.119086 12927 consensus_queue.cc:237] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.121235 12409 catalog_manager.cc:5495] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66). New cstate: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.126@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:02.200419 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.180029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:43895 (local address 127.0.0.1:48586)
0929 16:21:02.180324 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.180335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.180477 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.180725 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.180728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.181137 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:02.181228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.181234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.181968 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.181971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.182577 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.182585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.183003 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.183006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.183092 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.195669 (+ 12577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:02.195683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:02.197158 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:02.198753 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:02.198764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:02.198771 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:02.199006 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:02.199291 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:02.199294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:02.199296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:02.199377 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:02.199773 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:02.199781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:02.199892 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.200007 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.200175 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":122,"threads_started":1}
I20260929 16:21:02.200419 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.180070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.4.200.126:43895)
0929 16:21:02.180231 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.180235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.180420 (+ 185us) server_negotiation.cc:407] Connection header received
0929 16:21:02.180581 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.180590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.180628 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.180710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:02.181343 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.181853 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.182693 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.182899 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.183005 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.197329 (+ 14324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:02.197333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:02.197335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:02.197355 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:02.198648 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:02.199115 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:02.199119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:02.199120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:02.199193 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:02.199464 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:02.199470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:02.199472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:02.199653 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:02.199759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.200063 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.200231 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":59}
I20260929 16:21:02.204511 12409 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:48586:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:21:02.205178 12409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:02.209743 12687 tablet_service.cc:1467] Processing CreateTablet for tablet e3b5d16fd6264eae85bc91a0ba18f362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2c4c9102c548eb96a7c83f8d291631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:21:02.209761 12547 tablet_service.cc:1467] Processing CreateTablet for tablet e3b5d16fd6264eae85bc91a0ba18f362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2c4c9102c548eb96a7c83f8d291631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:21:02.210139 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b5d16fd6264eae85bc91a0ba18f362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.210142 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b5d16fd6264eae85bc91a0ba18f362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.209998 12815 tablet_service.cc:1467] Processing CreateTablet for tablet e3b5d16fd6264eae85bc91a0ba18f362 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e2c4c9102c548eb96a7c83f8d291631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:21:02.210346 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b5d16fd6264eae85bc91a0ba18f362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.212456 12921 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: Bootstrap starting.
I20260929 16:21:02.213037 12922 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: Bootstrap starting.
I20260929 16:21:02.213207 12921 tablet_bootstrap.cc:654] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.213987 12922 tablet_bootstrap.cc:654] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.214416 12921 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: No bootstrap required, opened a new log
I20260929 16:21:02.214504 12921 ts_tablet_manager.cc:1397] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:02.214547 12923 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: Bootstrap starting.
I20260929 16:21:02.214944 12921 raft_consensus.cc:348] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.215039 12921 raft_consensus.cc:374] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.215060 12921 raft_consensus.cc:729] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.215211 12921 consensus_queue.cc:260] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.215497 12921 ts_tablet_manager.cc:1428] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.215564 12923 tablet_bootstrap.cc:654] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.215691 12928 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.215827 12928 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.216924 12922 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: No bootstrap required, opened a new log
I20260929 16:21:02.216924 12923 tablet_bootstrap.cc:492] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: No bootstrap required, opened a new log
I20260929 16:21:02.217005 12923 ts_tablet_manager.cc:1397] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:21:02.217005 12922 ts_tablet_manager.cc:1397] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:21:02.217411 12922 raft_consensus.cc:348] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.217410 12923 raft_consensus.cc:348] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.217509 12922 raft_consensus.cc:374] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.217509 12923 raft_consensus.cc:374] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.217542 12922 raft_consensus.cc:729] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.217542 12923 raft_consensus.cc:729] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.217670 12923 consensus_queue.cc:260] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.217670 12922 consensus_queue.cc:260] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.217936 12922 ts_tablet_manager.cc:1428] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:21:02.217924 12923 ts_tablet_manager.cc:1428] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:21:02.217914 12929 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.217909 12927 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.218045 12929 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.218045 12927 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.370481 12927 raft_consensus.cc:482] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:02.370616 12927 raft_consensus.cc:504] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.371089 12927 leader_election.cc:288] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.371536 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b5d16fd6264eae85bc91a0ba18f362" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d" is_pre_election: true
I20260929 16:21:02.371573 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b5d16fd6264eae85bc91a0ba18f362" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62" is_pre_election: true
I20260929 16:21:02.371704 12565 raft_consensus.cc:2454] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 0.
I20260929 16:21:02.371719 12846 raft_consensus.cc:2454] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 0.
I20260929 16:21:02.372021 12638 leader_election.cc:302] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:02.372184 12927 raft_consensus.cc:2781] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:02.372241 12927 raft_consensus.cc:482] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:02.372280 12927 raft_consensus.cc:3037] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.373230 12927 raft_consensus.cc:504] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.373540 12927 leader_election.cc:288] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 election: Requested vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.373864 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b5d16fd6264eae85bc91a0ba18f362" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d"
I20260929 16:21:02.373977 12565 raft_consensus.cc:3037] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.373970 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b5d16fd6264eae85bc91a0ba18f362" candidate_uuid: "cdcab6fbd2034ef69033112033a2a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62"
I20260929 16:21:02.374121 12846 raft_consensus.cc:3037] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.375002 12565 raft_consensus.cc:2454] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 1.
I20260929 16:21:02.375113 12846 raft_consensus.cc:2454] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcab6fbd2034ef69033112033a2a3ee in term 1.
I20260929 16:21:02.375337 12638 leader_election.cc:302] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:02.375568 12927 raft_consensus.cc:2781] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:02.375701 12927 raft_consensus.cc:686] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [term 1 LEADER]: Becoming Leader. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Running, Role: LEADER
I20260929 16:21:02.375864 12927 consensus_queue.cc:237] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.376508 12953 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: TxnStatusTablet state changed. Reason: New leader cdcab6fbd2034ef69033112033a2a3ee. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.376616 12953 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:21:02.376904 12965 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:21:02.377110 12965 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:21:02.377696 12410 catalog_manager.cc:5495] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66). New cstate: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } health_report { overall_health: UNKNOWN } } }
I20260929 16:21:02.409376 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.405401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.4.200.66:37325)
0929 16:21:02.405633 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.405637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.405652 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:02.405759 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.405761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.405811 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.405908 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.406166 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.406776 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.407965 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.408203 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.408282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.408598 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.408998 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.409081 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.409200 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.409245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260929 16:21:02.409420 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.405282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.0.0.1:45900)
0929 16:21:02.405522 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.405537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.405636 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.405940 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.405943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.405951 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:21:02.406061 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.406066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.406882 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.406885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.407846 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.407857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.408308 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.408312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.408409 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.408460 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.409091 (+ 631us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.409107 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.409189 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.409279 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 16:21:02.412596 12846 raft_consensus.cc:1261] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Refusing update from remote peer cdcab6fbd2034ef69033112033a2a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.412698 12565 raft_consensus.cc:1261] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Refusing update from remote peer cdcab6fbd2034ef69033112033a2a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.413115 12927 consensus_queue.cc:1035] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.413295 12953 consensus_queue.cc:1035] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.414920 12929 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: TxnStatusTablet state changed. Reason: New leader cdcab6fbd2034ef69033112033a2a3ee. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.415055 12929 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.415180 12928 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: TxnStatusTablet state changed. Reason: New leader cdcab6fbd2034ef69033112033a2a3ee. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.415288 12928 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.417862 12953 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.417976 12953 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:21:02.418097 12953 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.418172 12953 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P cdcab6fbd2034ef69033112033a2a3ee: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:21:02.418365 12970 mvcc.cc:205] Tried to move back new op lower bound from 7334702540436504576 to 7334702540292698112. Current Snapshot: MvccSnapshot[applied={T|T < 7334702540436504576}]
I20260929 16:21:02.419268 12928 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.419369 12928 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P 525990dd76f8428bba9963cf78a89d2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.419725 12972 mvcc.cc:205] Tried to move back new op lower bound from 7334702540436504576 to 7334702540292698112. Current Snapshot: MvccSnapshot[applied={T|T < 7334702540436504576}]
I20260929 16:21:02.420380 12929 tablet_replica.cc:439] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdcab6fbd2034ef69033112033a2a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } } }
I20260929 16:21:02.420495 12929 tablet_replica.cc:442] T e3b5d16fd6264eae85bc91a0ba18f362 P d49b5ab2815448729d641f29a0abfc62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:21:02.428180 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.424728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.4.200.66:37325)
0929 16:21:02.424854 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.424859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.425015 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:21:02.425138 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.425140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.425179 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.425241 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.425492 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.425990 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.427221 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.427426 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.427498 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.427764 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.427828 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.427875 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.427992 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.428035 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:21:02.438365 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.434869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:35859 (local address 127.4.200.66:37325)
0929 16:21:02.435028 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.435032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.435168 (+ 136us) server_negotiation.cc:407] Connection header received
0929 16:21:02.435290 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.435292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.435328 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.435389 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.435648 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.436213 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.437418 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.437622 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.437698 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.437989 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.438044 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.438101 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.438206 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.438243 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:21:02.438704 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.434796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.4.200.66:35859)
0929 16:21:02.435083 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.435096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.435204 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.435417 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.435421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.435428 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:21:02.435556 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.435562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.436307 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.436311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.437294 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.437304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.437722 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.437726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.437818 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.437865 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.438115 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.438121 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.438192 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.438261 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":130,"threads_started":1}
I20260929 16:21:02.444805 12565 raft_consensus.cc:1261] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Refusing update from remote peer cdcab6fbd2034ef69033112033a2a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.445128 12846 raft_consensus.cc:1261] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Refusing update from remote peer cdcab6fbd2034ef69033112033a2a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.445184 12927 consensus_queue.cc:1035] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.445482 12927 consensus_queue.cc:1035] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.492938 12410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.493130 12410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.495705 12410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Sending DeleteTablet for 3 replicas of tablet c42b3606ffdf4d4e8d1b25baf0e1f682
I20260929 16:21:02.496304 12547 tablet_service.cc:1514] Processing DeleteTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:47590
I20260929 16:21:02.496320 12686 tablet_service.cc:1514] Processing DeleteTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:45888
I20260929 16:21:02.496620 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet c42b3606ffdf4d4e8d1b25baf0e1f682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260929 16:21:02.496944 12989 tablet_replica.cc:330] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: stopping tablet replica
I20260929 16:21:02.497073 12989 raft_consensus.cc:2229] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.497205 12989 raft_consensus.cc:2258] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.501317 12987 tablet_replica.cc:330] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: stopping tablet replica
I20260929 16:21:02.501474 12987 raft_consensus.cc:2229] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.501600 12987 raft_consensus.cc:2258] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.502966 12987 ts_tablet_manager.cc:1905] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.503453 12989 ts_tablet_manager.cc:1905] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.506102 12987 ts_tablet_manager.cc:1918] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.506191 12987 log.cc:1196] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/c42b3606ffdf4d4e8d1b25baf0e1f682
I20260929 16:21:02.506479 12987 ts_tablet_manager.cc:1939] T c42b3606ffdf4d4e8d1b25baf0e1f682 P 525990dd76f8428bba9963cf78a89d2d: Deleting consensus metadata
I20260929 16:21:02.508646 12989 ts_tablet_manager.cc:1918] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.508898 12989 log.cc:1196] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/c42b3606ffdf4d4e8d1b25baf0e1f682
I20260929 16:21:02.509506 12989 ts_tablet_manager.cc:1939] T c42b3606ffdf4d4e8d1b25baf0e1f682 P d49b5ab2815448729d641f29a0abfc62: Deleting consensus metadata
I20260929 16:21:02.510751 12394 catalog_manager.cc:4841] TS d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223): tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (table test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]) successfully deleted
I20260929 16:21:02.514565 12988 tablet_replica.cc:330] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: stopping tablet replica
I20260929 16:21:02.514703 12988 raft_consensus.cc:2229] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:21:02.514997 12988 raft_consensus.cc:2258] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.515705 12988 ts_tablet_manager.cc:1905] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.516546 12394 catalog_manager.cc:4841] TS 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697): tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (table test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]) successfully deleted
I20260929 16:21:02.518723 12988 ts_tablet_manager.cc:1918] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.518808 12988 log.cc:1196] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/c42b3606ffdf4d4e8d1b25baf0e1f682
I20260929 16:21:02.519075 12988 ts_tablet_manager.cc:1939] T c42b3606ffdf4d4e8d1b25baf0e1f682 P cdcab6fbd2034ef69033112033a2a3ee: Deleting consensus metadata
I20260929 16:21:02.519755 12397 catalog_manager.cc:4841] TS cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325): tablet c42b3606ffdf4d4e8d1b25baf0e1f682 (table test-table [id=91c8ea2081f9407c9b206b9b0bf8ef76]) successfully deleted
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:21:02.537982 12459 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:02.531550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.4.200.126:43895)
0929 16:21:02.531693 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.531698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.531831 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:21:02.531939 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.531942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.531999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.532098 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:02.532777 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.533584 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.534458 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.534800 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.534919 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.537852 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48592: BlockingRecv error: recv got EOF from 127.0.0.1:48592 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM
I20260929 16:21:02.547564 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.539407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.4.200.126:43895)
0929 16:21:02.539529 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.539533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.539604 (+ 71us) server_negotiation.cc:407] Connection header received
0929 16:21:02.539757 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.539760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.539799 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.539886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:02.540438 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.540926 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.541705 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.541920 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.542046 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.544373 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:02.544377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:02.544380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:02.544404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:02.545822 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:02.546254 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:02.546258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:02.546260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:02.546342 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:02.546681 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:02.546685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:02.546686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:02.546913 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:02.547044 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.547293 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.547405 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:21:02.550101 12410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:21:02.550607 12410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:02.555336 12547 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (DEFAULT_TABLE table=test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.555451 12686 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (DEFAULT_TABLE table=test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.555650 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.555744 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.555754 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (DEFAULT_TABLE table=test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.555995 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.557948 12921 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Bootstrap starting.
I20260929 16:21:02.558318 12923 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Bootstrap starting.
I20260929 16:21:02.559082 12921 tablet_bootstrap.cc:654] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.559158 12923 tablet_bootstrap.cc:654] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.559686 12922 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Bootstrap starting.
I20260929 16:21:02.560251 12923 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: No bootstrap required, opened a new log
I20260929 16:21:02.560318 12923 ts_tablet_manager.cc:1397] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:02.560571 12922 tablet_bootstrap.cc:654] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.560667 12923 raft_consensus.cc:348] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.560760 12923 raft_consensus.cc:374] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.560786 12923 raft_consensus.cc:729] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.560894 12923 consensus_queue.cc:260] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.561126 12923 ts_tablet_manager.cc:1428] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.561596 12921 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: No bootstrap required, opened a new log
I20260929 16:21:02.561674 12921 ts_tablet_manager.cc:1397] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:21:02.562007 12921 raft_consensus.cc:348] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.562107 12921 raft_consensus.cc:374] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.562135 12921 raft_consensus.cc:729] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.562237 12921 consensus_queue.cc:260] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.562505 12921 ts_tablet_manager.cc:1428] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.563592 12922 tablet_bootstrap.cc:492] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: No bootstrap required, opened a new log
I20260929 16:21:02.563694 12922 ts_tablet_manager.cc:1397] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:21:02.564138 12922 raft_consensus.cc:348] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.564267 12922 raft_consensus.cc:374] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.564299 12922 raft_consensus.cc:729] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.564410 12922 consensus_queue.cc:260] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.564662 12922 ts_tablet_manager.cc:1428] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.691609 12929 raft_consensus.cc:482] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:02.691762 12929 raft_consensus.cc:504] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.693127 12929 leader_election.cc:288] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325)
I20260929 16:21:02.697685 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.693772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:40607 (local address 127.4.200.65:41697)
0929 16:21:02.694087 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.694097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.694109 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:21:02.694180 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.694183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.694227 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.694314 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:02.694640 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.695365 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.696507 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.697001 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.697128 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.697178 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.697362 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.697437 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":65,"threads_started":1}
I20260929 16:21:02.698187 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.693384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:41697 (local address 127.4.200.67:40607)
0929 16:21:02.693764 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.693780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.693888 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.694330 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.694334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.694353 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:02.694504 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.694512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.695494 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.695497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.696389 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.696398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.697122 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.697126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.697234 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.697260 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.697331 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.697402 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":49,"thread_start_us":93,"threads_started":1}
I20260929 16:21:02.698686 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc2ec31d3a4ea2b331a9d805d64e8e" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d" is_pre_election: true
I20260929 16:21:02.698836 12565 raft_consensus.cc:2454] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49b5ab2815448729d641f29a0abfc62 in term 0.
I20260929 16:21:02.699060 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.693384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.4.200.67:33423)
0929 16:21:02.693830 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.693841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.693906 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.694189 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.694192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.694208 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:02.694337 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.694344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.695388 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.695394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.696756 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.696766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.698675 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.698679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.698774 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.698796 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.698864 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.698930 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":75,"threads_started":1}
I20260929 16:21:02.699169 12778 leader_election.cc:302] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, d49b5ab2815448729d641f29a0abfc62; no voters:
I20260929 16:21:02.699404 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.693484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:33423 (local address 127.4.200.66:37325)
0929 16:21:02.693608 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.693614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.693916 (+ 302us) server_negotiation.cc:407] Connection header received
0929 16:21:02.693992 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.693997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.694063 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.694172 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:02.694458 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.695265 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.696881 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.697487 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.697596 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.697641 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.699193 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.699287 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:21:02.699736 12929 raft_consensus.cc:2781] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:02.699795 12929 raft_consensus.cc:482] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:02.699831 12929 raft_consensus.cc:3037] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.701007 12929 raft_consensus.cc:504] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.701378 12929 leader_election.cc:288] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 election: Requested vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325)
I20260929 16:21:02.701696 12707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc2ec31d3a4ea2b331a9d805d64e8e" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee" is_pre_election: true
I20260929 16:21:02.701829 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc2ec31d3a4ea2b331a9d805d64e8e" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d"
I20260929 16:21:02.701869 12707 raft_consensus.cc:2454] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49b5ab2815448729d641f29a0abfc62 in term 0.
I20260929 16:21:02.701936 12565 raft_consensus.cc:3037] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.702106 12706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc2ec31d3a4ea2b331a9d805d64e8e" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee"
I20260929 16:21:02.702200 12706 raft_consensus.cc:3037] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.703130 12565 raft_consensus.cc:2454] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49b5ab2815448729d641f29a0abfc62 in term 1.
I20260929 16:21:02.703413 12706 raft_consensus.cc:2454] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49b5ab2815448729d641f29a0abfc62 in term 1.
I20260929 16:21:02.703456 12778 leader_election.cc:302] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, d49b5ab2815448729d641f29a0abfc62; no voters:
I20260929 16:21:02.703644 12929 raft_consensus.cc:2781] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:02.703907 12929 raft_consensus.cc:686] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 1 LEADER]: Becoming Leader. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Running, Role: LEADER
I20260929 16:21:02.704150 12929 consensus_queue.cc:237] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.706609 12409 catalog_manager.cc:5495] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 reported cstate change: term changed from 0 to 1, leader changed from <none> to d49b5ab2815448729d641f29a0abfc62 (127.4.200.67). New cstate: current_term: 1 leader_uuid: "d49b5ab2815448729d641f29a0abfc62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } health_report { overall_health: HEALTHY } } }
I20260929 16:21:02.722222 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.718270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.4.200.67:44223)
0929 16:21:02.718511 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.718515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.718530 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:02.718600 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.718602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.718654 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.718761 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.718988 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.719492 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.720672 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.720899 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.720998 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.721235 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.721698 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.721805 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.721923 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.721995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260929 16:21:02.724951 12706 raft_consensus.cc:1261] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Refusing update from remote peer d49b5ab2815448729d641f29a0abfc62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.724951 12565 raft_consensus.cc:1261] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Refusing update from remote peer d49b5ab2815448729d641f29a0abfc62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.725461 12929 consensus_queue.cc:1035] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.725709 13008 consensus_queue.cc:1035] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.736089 12409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.736260 12409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.738473 12409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Sending DeleteTablet for 3 replicas of tablet 54bc2ec31d3a4ea2b331a9d805d64e8e
I20260929 16:21:02.739049 12547 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:47590
I20260929 16:21:02.739080 12686 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:45888
I20260929 16:21:02.739320 12987 tablet_replica.cc:330] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: stopping tablet replica
I20260929 16:21:02.739477 12987 raft_consensus.cc:2229] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.739627 12987 raft_consensus.cc:2258] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.739825 12988 tablet_replica.cc:330] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: stopping tablet replica
I20260929 16:21:02.739957 12988 raft_consensus.cc:2229] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.740089 12988 raft_consensus.cc:2258] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.740783 12987 ts_tablet_manager.cc:1905] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.740855 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc2ec31d3a4ea2b331a9d805d64e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260929 16:21:02.741008 12989 tablet_replica.cc:330] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: stopping tablet replica
I20260929 16:21:02.741128 12989 raft_consensus.cc:2229] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:21:02.741303 12988 ts_tablet_manager.cc:1905] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.741416 12989 raft_consensus.cc:2258] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.742069 12989 ts_tablet_manager.cc:1905] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.744028 12988 ts_tablet_manager.cc:1918] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:02.744115 12988 log.cc:1196] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/54bc2ec31d3a4ea2b331a9d805d64e8e
I20260929 16:21:02.744321 12987 ts_tablet_manager.cc:1918] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:02.744403 12987 log.cc:1196] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/54bc2ec31d3a4ea2b331a9d805d64e8e
I20260929 16:21:02.744503 12988 ts_tablet_manager.cc:1939] T 54bc2ec31d3a4ea2b331a9d805d64e8e P cdcab6fbd2034ef69033112033a2a3ee: Deleting consensus metadata
I20260929 16:21:02.744526 12989 ts_tablet_manager.cc:1918] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:02.744594 12989 log.cc:1196] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/54bc2ec31d3a4ea2b331a9d805d64e8e
I20260929 16:21:02.744678 12987 ts_tablet_manager.cc:1939] T 54bc2ec31d3a4ea2b331a9d805d64e8e P 525990dd76f8428bba9963cf78a89d2d: Deleting consensus metadata
I20260929 16:21:02.744839 12989 ts_tablet_manager.cc:1939] T 54bc2ec31d3a4ea2b331a9d805d64e8e P d49b5ab2815448729d641f29a0abfc62: Deleting consensus metadata
I20260929 16:21:02.745034 12409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:21:02.745493 12409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:02.745993 12394 catalog_manager.cc:4841] TS d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223): tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (table test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]) successfully deleted
I20260929 16:21:02.746475 12397 catalog_manager.cc:4841] TS cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325): tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (table test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]) successfully deleted
I20260929 16:21:02.747179 12394 catalog_manager.cc:4841] TS 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697): tablet 54bc2ec31d3a4ea2b331a9d805d64e8e (table test-table [id=db1f53f35d8c4795b75b53fcdd1a78db]) successfully deleted
I20260929 16:21:02.750375 12547 tablet_service.cc:1467] Processing CreateTablet for tablet a7f29dade36549a6967e1b64bcf74def (DEFAULT_TABLE table=test-table [id=7bfd27d8f0b5452f8b5b1dd9f30206fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.750568 12815 tablet_service.cc:1467] Processing CreateTablet for tablet a7f29dade36549a6967e1b64bcf74def (DEFAULT_TABLE table=test-table [id=7bfd27d8f0b5452f8b5b1dd9f30206fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.750641 12686 tablet_service.cc:1467] Processing CreateTablet for tablet a7f29dade36549a6967e1b64bcf74def (DEFAULT_TABLE table=test-table [id=7bfd27d8f0b5452f8b5b1dd9f30206fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:02.750701 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f29dade36549a6967e1b64bcf74def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.750794 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f29dade36549a6967e1b64bcf74def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.750921 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f29dade36549a6967e1b64bcf74def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:02.752997 12922 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Bootstrap starting.
I20260929 16:21:02.753001 12921 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Bootstrap starting.
I20260929 16:21:02.753396 12923 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Bootstrap starting.
I20260929 16:21:02.753806 12921 tablet_bootstrap.cc:654] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.753825 12922 tablet_bootstrap.cc:654] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.754227 12923 tablet_bootstrap.cc:654] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:02.755345 12922 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: No bootstrap required, opened a new log
I20260929 16:21:02.755427 12922 ts_tablet_manager.cc:1397] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:21:02.755699 12921 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: No bootstrap required, opened a new log
I20260929 16:21:02.755733 12923 tablet_bootstrap.cc:492] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: No bootstrap required, opened a new log
I20260929 16:21:02.755771 12921 ts_tablet_manager.cc:1397] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:21:02.755811 12923 ts_tablet_manager.cc:1397] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:21:02.755878 12922 raft_consensus.cc:348] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.755980 12922 raft_consensus.cc:374] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.756012 12922 raft_consensus.cc:729] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49b5ab2815448729d641f29a0abfc62, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.756124 12922 consensus_queue.cc:260] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.756160 12921 raft_consensus.cc:348] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.756253 12921 raft_consensus.cc:374] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.756282 12921 raft_consensus.cc:729] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.756281 12923 raft_consensus.cc:348] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.756390 12922 ts_tablet_manager.cc:1428] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.756390 12923 raft_consensus.cc:374] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:02.756392 12921 consensus_queue.cc:260] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.756429 12923 raft_consensus.cc:729] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcab6fbd2034ef69033112033a2a3ee, State: Initialized, Role: FOLLOWER
I20260929 16:21:02.756572 12923 consensus_queue.cc:260] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.756632 12921 ts_tablet_manager.cc:1428] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.756811 12923 ts_tablet_manager.cc:1428] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:21:02.793193 12928 raft_consensus.cc:482] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:02.793350 12928 raft_consensus.cc:504] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.793795 12928 leader_election.cc:288] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.794235 12706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee" is_pre_election: true
I20260929 16:21:02.794253 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62" is_pre_election: true
I20260929 16:21:02.794353 12929 raft_consensus.cc:482] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:02.794420 12706 raft_consensus.cc:2454] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 0.
I20260929 16:21:02.794449 12929 raft_consensus.cc:504] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.794749 12501 leader_election.cc:302] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:02.794770 12846 raft_consensus.cc:2454] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 0.
I20260929 16:21:02.794822 12929 leader_election.cc:288] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325)
I20260929 16:21:02.794914 12928 raft_consensus.cc:2781] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:02.794961 12928 raft_consensus.cc:482] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:02.794998 12928 raft_consensus.cc:3037] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.795213 12706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee" is_pre_election: true
I20260929 16:21:02.795334 12706 raft_consensus.cc:2454] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49b5ab2815448729d641f29a0abfc62 in term 0.
I20260929 16:21:02.795650 12781 leader_election.cc:302] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdcab6fbd2034ef69033112033a2a3ee, d49b5ab2815448729d641f29a0abfc62; no voters:
I20260929 16:21:02.795818 12929 raft_consensus.cc:2781] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:02.795872 12929 raft_consensus.cc:482] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:02.795926 12929 raft_consensus.cc:3037] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.796161 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d" is_pre_election: true
I20260929 16:21:02.796231 12928 raft_consensus.cc:504] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.796401 12565 raft_consensus.cc:2379] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d49b5ab2815448729d641f29a0abfc62 in current term 1: Already voted for candidate 525990dd76f8428bba9963cf78a89d2d in this term.
I20260929 16:21:02.796545 12928 leader_election.cc:288] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 election: Requested vote from peers cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325), d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223)
I20260929 16:21:02.796936 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49b5ab2815448729d641f29a0abfc62"
I20260929 16:21:02.797144 12929 raft_consensus.cc:504] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.797307 12706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "525990dd76f8428bba9963cf78a89d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee"
I20260929 16:21:02.797408 12706 raft_consensus.cc:3037] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:02.797482 12929 leader_election.cc:288] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 election: Requested vote from peers 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697), cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325)
I20260929 16:21:02.797773 12565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525990dd76f8428bba9963cf78a89d2d"
I20260929 16:21:02.797829 12707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f29dade36549a6967e1b64bcf74def" candidate_uuid: "d49b5ab2815448729d641f29a0abfc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcab6fbd2034ef69033112033a2a3ee"
I20260929 16:21:02.797904 12565 raft_consensus.cc:2379] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d49b5ab2815448729d641f29a0abfc62 in current term 1: Already voted for candidate 525990dd76f8428bba9963cf78a89d2d in this term.
I20260929 16:21:02.798221 12778 leader_election.cc:302] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49b5ab2815448729d641f29a0abfc62; no voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee
I20260929 16:21:02.798290 12846 raft_consensus.cc:2379] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 525990dd76f8428bba9963cf78a89d2d in current term 1: Already voted for candidate d49b5ab2815448729d641f29a0abfc62 in this term.
I20260929 16:21:02.798548 12929 raft_consensus.cc:2726] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 16:21:02.798543 12706 raft_consensus.cc:2454] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525990dd76f8428bba9963cf78a89d2d in term 1.
I20260929 16:21:02.798790 12501 leader_election.cc:302] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525990dd76f8428bba9963cf78a89d2d, cdcab6fbd2034ef69033112033a2a3ee; no voters:
I20260929 16:21:02.798967 12928 raft_consensus.cc:2781] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:02.799161 12928 raft_consensus.cc:686] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 LEADER]: Becoming Leader. State: Replica: 525990dd76f8428bba9963cf78a89d2d, State: Running, Role: LEADER
I20260929 16:21:02.799320 12928 consensus_queue.cc:237] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } }
I20260929 16:21:02.800936 12410 catalog_manager.cc:5495] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 525990dd76f8428bba9963cf78a89d2d (127.4.200.65). New cstate: current_term: 1 leader_uuid: "525990dd76f8428bba9963cf78a89d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525990dd76f8428bba9963cf78a89d2d" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 } health_report { overall_health: UNKNOWN } } }
I20260929 16:21:02.819591 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.816354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.4.200.65:41697)
0929 16:21:02.816455 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.816462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.816481 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:21:02.816588 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.816590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.816636 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.816697 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.816948 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.817453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.818627 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.818834 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.818914 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.819173 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.819238 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.819291 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.819403 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.819447 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 16:21:02.830024 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.826663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37325 (local address 127.4.200.65:49797)
0929 16:21:02.826997 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.827010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.827135 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.827403 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.827406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.827414 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:21:02.827514 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.827519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.828235 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.828241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.828851 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.828859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.829307 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.829311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.829396 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.829447 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.829738 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.829747 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.829812 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.829887 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":112,"threads_started":1}
I20260929 16:21:02.830024 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.826768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:49797 (local address 127.4.200.66:37325)
0929 16:21:02.826900 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.826905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.827085 (+ 180us) server_negotiation.cc:407] Connection header received
0929 16:21:02.827246 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.827249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.827305 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.827383 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.827622 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.828132 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.828972 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.829197 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.829307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.829583 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.829654 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.829734 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.829821 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.829874 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:21:02.835330 12846 raft_consensus.cc:1261] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Refusing update from remote peer 525990dd76f8428bba9963cf78a89d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.835367 12706 raft_consensus.cc:1261] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Refusing update from remote peer 525990dd76f8428bba9963cf78a89d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:02.835686 12928 consensus_queue.cc:1035] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49b5ab2815448729d641f29a0abfc62" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.835989 12928 consensus_queue.cc:1035] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdcab6fbd2034ef69033112033a2a3ee" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:02.855470 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.851445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:41697 (local address 127.4.200.66:56845)
0929 16:21:02.851674 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.851690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:02.851766 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:02.852263 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:02.852265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:02.852272 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:21:02.852357 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.852361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.853366 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.853369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.854150 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:02.854160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.854827 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.854830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:02.854914 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.854945 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.855205 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.855213 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:21:02.855277 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:21:02.855355 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 16:21:02.855470 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.851905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:56845 (local address 127.4.200.65:41697)
0929 16:21:02.852037 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:02.852042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:02.852059 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:21:02.852102 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:02.852105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:02.852153 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:02.852234 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:02.852463 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.853251 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.854497 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:02.854719 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:02.854831 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:02.855046 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:02.855123 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:02.855201 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:02.855287 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 16:21:02.855332 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:21:02.878607 12408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.878796 12408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:02.881079 12408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f88e6058e13488d97045106e9debc58: Sending DeleteTablet for 3 replicas of tablet a7f29dade36549a6967e1b64bcf74def
I20260929 16:21:02.881664 12547 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f29dade36549a6967e1b64bcf74def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:47590
I20260929 16:21:02.881664 12686 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f29dade36549a6967e1b64bcf74def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:45888
I20260929 16:21:02.881851 12987 tablet_replica.cc:330] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: stopping tablet replica
I20260929 16:21:02.881850 12988 tablet_replica.cc:330] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: stopping tablet replica
I20260929 16:21:02.881975 12988 raft_consensus.cc:2229] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.881974 12987 raft_consensus.cc:2229] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:21:02.882091 12988 raft_consensus.cc:2258] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.882195 12987 raft_consensus.cc:2258] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.882686 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f29dade36549a6967e1b64bcf74def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:02 UTC) from {username='oryx', principal='oryx/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260929 16:21:02.882831 12988 ts_tablet_manager.cc:1905] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.882881 12989 tablet_replica.cc:330] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: stopping tablet replica
I20260929 16:21:02.882969 12989 raft_consensus.cc:2229] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:02.883050 12989 raft_consensus.cc:2258] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:02.883837 12987 ts_tablet_manager.cc:1905] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.883838 12989 ts_tablet_manager.cc:1905] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:02.885638 12988 ts_tablet_manager.cc:1918] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.885711 12988 log.cc:1196] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/a7f29dade36549a6967e1b64bcf74def
I20260929 16:21:02.885964 12988 ts_tablet_manager.cc:1939] T a7f29dade36549a6967e1b64bcf74def P cdcab6fbd2034ef69033112033a2a3ee: Deleting consensus metadata
I20260929 16:21:02.886175 12989 ts_tablet_manager.cc:1918] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.886207 12987 ts_tablet_manager.cc:1918] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:21:02.886247 12989 log.cc:1196] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/a7f29dade36549a6967e1b64bcf74def
I20260929 16:21:02.886494 12987 log.cc:1196] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/a7f29dade36549a6967e1b64bcf74def
I20260929 16:21:02.886767 12987 ts_tablet_manager.cc:1939] T a7f29dade36549a6967e1b64bcf74def P 525990dd76f8428bba9963cf78a89d2d: Deleting consensus metadata
I20260929 16:21:02.886926 12397 catalog_manager.cc:4841] TS cdcab6fbd2034ef69033112033a2a3ee (127.4.200.66:37325): tablet a7f29dade36549a6967e1b64bcf74def (table test-table [id=7bfd27d8f0b5452f8b5b1dd9f30206fb]) successfully deleted
I20260929 16:21:02.887256 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 12482
I20260929 16:21:02.889457 12989 ts_tablet_manager.cc:1939] T a7f29dade36549a6967e1b64bcf74def P d49b5ab2815448729d641f29a0abfc62: Deleting consensus metadata
I20260929 16:21:02.890105 12394 catalog_manager.cc:4841] TS d49b5ab2815448729d641f29a0abfc62 (127.4.200.67:44223): tablet a7f29dade36549a6967e1b64bcf74def (table test-table [id=7bfd27d8f0b5452f8b5b1dd9f30206fb]) successfully deleted
W20260929 16:21:02.895684 12394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:41697 (error 108)
I20260929 16:21:02.895807 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 12622
I20260929 16:21:02.897464 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:02.897050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:41697 (local address 127.0.0.1:47612)
0929 16:21:02.897255 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:02.897334 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260929 16:21:02.898043 12394 catalog_manager.cc:4584] TS 525990dd76f8428bba9963cf78a89d2d (127.4.200.65:41697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7f29dade36549a6967e1b64bcf74def: Network error: Client connection negotiation failed: client connection to 127.4.200.65:41697: connect: Connection refused (error 111)
I20260929 16:21:02.904480 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 12762
I20260929 16:21:02.911161 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 12378
2026-09-29T16:21:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3337 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[13031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[13031](info): set up 2 sockets
Sep 29 16:21:02 dist-test-slave-s9ct krb5kdc[13031](info): commencing operation
krb5kdc: starting...
W20260929 16:21:03.951279 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:21:04 dist-test-slave-s9ct krb5kdc[13031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:21:04.078624 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:37741
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:04.191258 13044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:04.191566 13044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:04.191679 13044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:04.195724 13044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:04.195799 13044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:04.195842 13044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:04.195899 13044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:04.195935 13044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:04.202354 13044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:04.203475 13044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:04.204515 13044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:04.213055 13052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:04.213214 13050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:04.213316 13044 server_base.cc:1034] running on GCE node
W20260929 16:21:04.213055 13049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:04.213920 13044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:04.215116 13044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:04.216276 13044 hybrid_clock.cc:648] HybridClock initialized: now 1790698864216261 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:21:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1315 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:21:04 dist-test-slave-s9ct krb5kdc[13057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:21:04 dist-test-slave-s9ct krb5kdc[13057](info): set up 2 sockets
Sep 29 16:21:04 dist-test-slave-s9ct krb5kdc[13057](info): commencing operation
krb5kdc: starting...
W20260929 16:21:05.271473 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:21:05 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:21:05.389240 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36037
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:05.500836 13070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:05.501150 13070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:05.501235 13070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:05.505352 13070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:05.505460 13070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:05.505499 13070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:05.505538 13070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:05.505574 13070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:05.511101 13070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:05.512256 13070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:05.513325 13070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:05.522265 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:05.522298 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:05.522440 13070 server_base.cc:1034] running on GCE node
W20260929 16:21:05.522274 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:05.523072 13070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:05.524139 13070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:05.525318 13070 hybrid_clock.cc:648] HybridClock initialized: now 1790698865525289 us; error 41 us; skew 500 ppm
Sep 29 16:21:05 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:05.528136 13070 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:21:05.529102 13070 webserver.cc:460] Webserver started at http://127.4.200.126:46707/ using document root <none> and password file <none>
I20260929 16:21:05.529698 13070 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:05.529763 13070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:05.529984 13070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:05.532150 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "ce94c6ebfcd549ffb057d98f094a92be"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "1315d8fefa1f54a4939d49ebcf65cb40"
server_key_iv: "a1ce6fa6e48195d1e76966b17349ad93"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.532756 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "ce94c6ebfcd549ffb057d98f094a92be"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "1315d8fefa1f54a4939d49ebcf65cb40"
server_key_iv: "a1ce6fa6e48195d1e76966b17349ad93"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.537153 13070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 16:21:05.539813 13085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:05.541072 13070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:21:05.541217 13070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ce94c6ebfcd549ffb057d98f094a92be"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "1315d8fefa1f54a4939d49ebcf65cb40"
server_key_iv: "a1ce6fa6e48195d1e76966b17349ad93"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.541340 13070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:05.552243 13070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:05.555585 13070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:05.555807 13070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:05.563532 13070 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36037
I20260929 16:21:05.563560 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36037 every 8 connection(s)
I20260929 16:21:05.564622 13070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:05.565487 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13070
I20260929 16:21:05.565629 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:21:05.565964 4897 external_mini_cluster.cc:1428] Setting key 393ff2d4d0357e8eb9b763c1e54fe16a
I20260929 16:21:05.568059 13138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:05.574919 13138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Bootstrap starting.
Sep 29 16:21:05 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:05.577354 13138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:05.578256 13138 log.cc:824] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:05.580467 13138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: No bootstrap required, opened a new log
I20260929 16:21:05.580660 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:05.567392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.4.200.126:36037)
0929 16:21:05.567817 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:05.567833 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:05.567868 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:21:05.568555 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:05.568572 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:05.568906 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:05.569203 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:05.570103 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.571157 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.572309 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.572763 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.572880 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:05.575785 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:05.575790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:05.575804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:05.575832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:05.577608 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:05.578080 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:05.578085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:05.578090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:05.578206 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:05.578577 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:05.578579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:05.578580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:05.579123 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:05.579288 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:05.579561 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:21:05.579869 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
I20260929 16:21:05.584082 13138 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:05.584329 13138 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:05.584407 13138 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce94c6ebfcd549ffb057d98f094a92be, State: Initialized, Role: FOLLOWER
I20260929 16:21:05.584806 13138 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:05.584923 13138 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:05.585011 13138 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:05.585100 13138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:05.585999 13138 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:05.586349 13138 leader_election.cc:302] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce94c6ebfcd549ffb057d98f094a92be; no voters:
I20260929 16:21:05.586726 13138 leader_election.cc:288] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:21:05.586838 13143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:05.587049 13143 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 LEADER]: Becoming Leader. State: Replica: ce94c6ebfcd549ffb057d98f094a92be, State: Running, Role: LEADER
I20260929 16:21:05.587373 13143 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:05.587994 13138 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:05.589030 13145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce94c6ebfcd549ffb057d98f094a92be. Latest consensus state: current_term: 1 leader_uuid: "ce94c6ebfcd549ffb057d98f094a92be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } } }
I20260929 16:21:05.589165 13145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:05.589541 13144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce94c6ebfcd549ffb057d98f094a92be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } } }
I20260929 16:21:05.590140 13144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:05.590584 13152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:05.593165 13152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:05.598971 13152 catalog_manager.cc:1347] Generated new cluster ID: 918e2f31b1994703aabe7ea16cbc479d
I20260929 16:21:05.599052 13152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:05.616339 13152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:21:05.617374 13152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:05.625571 13152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Generated new TSK 0
I20260929 16:21:05.626330 13152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:21:05.662032 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:05.774610 13164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:05.774870 13164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:05.774912 13164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:05.778908 13164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:05.778959 13164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:05.779016 13164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:05.784134 13164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:05.785115 13164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:05.786101 13164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:05.795917 13170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:05.795970 13169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:05.796296 13172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:05.796772 13164 server_base.cc:1034] running on GCE node
I20260929 16:21:05.797179 13164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:05.797900 13164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:05.799075 13164 hybrid_clock.cc:648] HybridClock initialized: now 1790698865799055 us; error 33 us; skew 500 ppm
Sep 29 16:21:05 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:05.801887 13164 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:21:05.802915 13164 webserver.cc:460] Webserver started at http://127.4.200.65:46849/ using document root <none> and password file <none>
I20260929 16:21:05.803505 13164 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:05.803642 13164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:05.803882 13164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:05.805989 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "fe9193f9f0764369a49f30a6fc98e0ef"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "84fedb8f237b272eb50fd9202ed24b6c"
server_key_iv: "e79c4fe9d38ff3d998c645fe9544c550"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.806622 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "fe9193f9f0764369a49f30a6fc98e0ef"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "84fedb8f237b272eb50fd9202ed24b6c"
server_key_iv: "e79c4fe9d38ff3d998c645fe9544c550"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.810621 13164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:21:05.812990 13179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:05.814220 13164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:05.814478 13164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "fe9193f9f0764369a49f30a6fc98e0ef"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "84fedb8f237b272eb50fd9202ed24b6c"
server_key_iv: "e79c4fe9d38ff3d998c645fe9544c550"
server_key_version: "encryptionkey@0"
I20260929 16:21:05.814597 13164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:05.843257 13164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:05.846529 13164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:05.846731 13164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:05.847342 13164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:05.848192 13164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:05.848263 13164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:05.848335 13164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:05.848380 13164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:05.857331 13164 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:45035
I20260929 16:21:05.857352 13292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:45035 every 8 connection(s)
I20260929 16:21:05.858234 13164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:05.858877 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13164
I20260929 16:21:05.859017 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:21:05.859331 4897 external_mini_cluster.cc:1428] Setting key aed4f1a509510d049f25f30a04f86146
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Sep 29 16:21:05 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:21:05.873997 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:05.861221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:46289 (local address 127.4.200.126:36037)
0929 16:21:05.861380 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:05.861385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:05.862326 (+ 941us) server_negotiation.cc:407] Connection header received
0929 16:21:05.863724 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:05.863727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:05.863770 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:05.863835 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:05.865508 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.866040 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.867273 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.867508 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.867625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:05.870134 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:05.870137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:05.870140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:05.870166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:05.871638 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:05.872235 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:05.872242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:05.872247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:05.872377 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:05.872785 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:05.872791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:05.872794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:05.872997 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:05.873108 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:05.873680 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 16:21:05.873826 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:21:05.874964 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:05.861635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:46289)
0929 16:21:05.862141 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:05.862211 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:05.863495 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:05.864019 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:05.864031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:05.864582 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:05.865345 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:05.865369 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.866155 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.866158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:05.867126 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:05.867136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:05.867679 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:05.867683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:05.867859 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:05.868495 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:05.868511 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:05.869946 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:05.871765 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:05.871771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:05.871788 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:05.872118 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:05.872509 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:05.872512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:05.872515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:05.872676 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:05.873120 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:05.873129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:05.873425 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 16:21:05.873638 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 16:21:05.873989 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
I20260929 16:21:05.876405 13293 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36037
I20260929 16:21:05.876704 13293 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:05.877413 13293 heartbeater.cc:511] Master 127.4.200.126:36037 requested a full tablet report, sending...
I20260929 16:21:05.879467 13102 ts_manager.cc:194] Registered new tserver with Master: fe9193f9f0764369a49f30a6fc98e0ef (127.4.200.65:45035)
I20260929 16:21:05.881286 13102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:46289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:21:05.890728 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:06.002976 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:06.003240 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:06.003320 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:06.007437 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:06.007499 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:06.007615 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:06.012848 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:06.013897 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:06.014972 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:06.024521 13304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.024528 13306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.024683 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.024705 13298 server_base.cc:1034] running on GCE node
I20260929 16:21:06.025141 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:06.025748 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:06.026938 13298 hybrid_clock.cc:648] HybridClock initialized: now 1790698866026927 us; error 35 us; skew 500 ppm
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:06.029631 13298 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:21:06.030591 13298 webserver.cc:460] Webserver started at http://127.4.200.66:41439/ using document root <none> and password file <none>
I20260929 16:21:06.031162 13298 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:06.031251 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:06.031486 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:06.033478 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "39d68d414f334c64892bca8ba15e5fe0"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "453e377511d97873db9e506227d30a93"
server_key_iv: "42cd111c44ee995a098e6d003c3fed33"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.034017 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "39d68d414f334c64892bca8ba15e5fe0"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "453e377511d97873db9e506227d30a93"
server_key_iv: "42cd111c44ee995a098e6d003c3fed33"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.037635 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:21:06.040167 13313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.041339 13298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:06.041492 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "39d68d414f334c64892bca8ba15e5fe0"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "453e377511d97873db9e506227d30a93"
server_key_iv: "42cd111c44ee995a098e6d003c3fed33"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.041603 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:06.074007 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:06.076922 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:06.077145 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:06.077754 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:06.078599 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:06.078670 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.078739 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:06.078787 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.088238 13298 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:36053
I20260929 16:21:06.088285 13426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:36053 every 8 connection(s)
I20260929 16:21:06.089260 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:06.097594 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13298
I20260929 16:21:06.097730 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:21:06.098100 4897 external_mini_cluster.cc:1428] Setting key 6f141d5f3bf35259f1b47a480df920b9
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:06.104259 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.091230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:36901 (local address 127.4.200.126:36037)
0929 16:21:06.091395 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:06.091400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:06.092273 (+ 873us) server_negotiation.cc:407] Connection header received
0929 16:21:06.093508 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:06.093511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:06.093559 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:06.093654 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:06.094945 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.095469 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.096680 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.096929 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.097026 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.100304 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:06.100307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:06.100309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:06.100333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:06.101847 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:06.102541 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:06.102544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:06.102545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:06.102644 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:06.103056 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:06.103059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:06.103060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:06.103233 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:06.103306 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:06.104018 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 16:21:06.104126 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:21:06.105599 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.091687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:36901)
0929 16:21:06.092116 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:06.092159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:06.093299 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:06.093828 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:06.093836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:06.094193 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:06.094775 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.094791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.095587 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.095590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.096524 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.096534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.097094 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.097097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.097295 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.098074 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:06.098096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:06.100106 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:06.101970 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:06.101979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:06.102000 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:06.102385 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:06.102747 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:06.102752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:06.102757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:06.102957 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:06.103337 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:06.103346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:06.103714 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 16:21:06.103949 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 16:21:06.104363 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260929 16:21:06.106881 13427 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36037
I20260929 16:21:06.107151 13427 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:06.107662 13427 heartbeater.cc:511] Master 127.4.200.126:36037 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:21:06.108754 13102 ts_manager.cc:194] Registered new tserver with Master: 39d68d414f334c64892bca8ba15e5fe0 (127.4.200.66:36053)
I20260929 16:21:06.109351 13102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:36901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:21:06.125751 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:06.239790 13432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:06.240048 13432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:06.240089 13432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:06.244091 13432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:06.244153 13432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:06.244261 13432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:06.249805 13432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:06.250944 13432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:06.251995 13432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:06.262225 13438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.262251 13437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.262547 13432 server_base.cc:1034] running on GCE node
W20260929 16:21:06.262246 13440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.263115 13432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:06.263821 13432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:06.265010 13432 hybrid_clock.cc:648] HybridClock initialized: now 1790698866264967 us; error 56 us; skew 500 ppm
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:06.268085 13432 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:21:06.268997 13432 webserver.cc:460] Webserver started at http://127.4.200.67:36253/ using document root <none> and password file <none>
I20260929 16:21:06.269579 13432 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:06.269668 13432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:06.269910 13432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:06.272060 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "631b2d38556e432080a2b8fd746f4b0a"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "cae0fa7b7a4dfe7b10d50a9cf2357deb"
server_key_iv: "6933fdf838cbb34a5f2117bcdb5abc1c"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.272619 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "631b2d38556e432080a2b8fd746f4b0a"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "cae0fa7b7a4dfe7b10d50a9cf2357deb"
server_key_iv: "6933fdf838cbb34a5f2117bcdb5abc1c"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.276451 13432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:21:06.278865 13447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.280081 13432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:06.280221 13432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "631b2d38556e432080a2b8fd746f4b0a"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "cae0fa7b7a4dfe7b10d50a9cf2357deb"
server_key_iv: "6933fdf838cbb34a5f2117bcdb5abc1c"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.280335 13432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:06.298898 13432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:06.302227 13432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:06.302520 13432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:06.303146 13432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:06.304023 13432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:06.304095 13432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.304168 13432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:06.304236 13432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.312984 13432 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:40135
I20260929 16:21:06.313006 13560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:40135 every 8 connection(s)
I20260929 16:21:06.313874 13432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:06.322328 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13432
I20260929 16:21:06.322538 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:21:06.322857 4897 external_mini_cluster.cc:1428] Setting key e0cad0515067d4513aff20b6d81f57c1
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:06.327669 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.315681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:49635 (local address 127.4.200.126:36037)
0929 16:21:06.315837 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:06.315842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:06.316723 (+ 881us) server_negotiation.cc:407] Connection header received
0929 16:21:06.318002 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:06.318006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:06.318053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:06.318127 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:06.319363 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.319980 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.321037 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.321244 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.321319 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.324161 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:06.324165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:06.324168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:06.324210 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:06.325803 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:06.326273 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:06.326276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:06.326277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:06.326350 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:06.326699 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:06.326701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:06.326702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:06.326884 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:06.326961 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:06.327442 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 16:21:06.327543 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:21:06.328537 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.316124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:49635)
0929 16:21:06.316551 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:06.316592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:06.317813 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:06.318291 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:06.318298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:06.318772 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:06.319239 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.319253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.320085 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.320087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.320938 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.320946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.321366 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.321368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.321502 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.322146 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:06.322161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:06.323941 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:06.325904 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:06.325908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:06.325921 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:06.326185 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:06.326469 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:06.326471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:06.326473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:06.326613 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:06.326979 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:06.326985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:06.327236 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 16:21:06.327395 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:21:06.327658 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260929 16:21:06.329706 13561 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36037
I20260929 16:21:06.329943 13561 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:06.330499 13561 heartbeater.cc:511] Master 127.4.200.126:36037 requested a full tablet report, sending...
I20260929 16:21:06.331519 13102 ts_manager.cc:194] Registered new tserver with Master: 631b2d38556e432080a2b8fd746f4b0a (127.4.200.67:40135)
I20260929 16:21:06.332091 13102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:49635
I20260929 16:21:06.336843 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
Entry for principal oryx/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab.
WARNING: no policy specified for oryx/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
Entry for principal oryx/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab.
WARNING: no policy specified for oryx/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
Entry for principal oryx/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab.
I20260929 16:21:06.416908 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13164
I20260929 16:21:06.422582 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13298
I20260929 16:21:06.428086 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13432
I20260929 16:21:06.433462 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13070
I20260929 16:21:06.439625 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36037
--webserver_interface=127.4.200.126
--webserver_port=46707
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:06.553290 13570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:06.553575 13570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:06.553618 13570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:06.557768 13570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:06.557835 13570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:06.557857 13570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:06.557873 13570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:06.557911 13570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:06.563375 13570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:06.564445 13570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:06.565467 13570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:06.573891 13575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.573891 13578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.573897 13576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.574316 13570 server_base.cc:1034] running on GCE node
I20260929 16:21:06.574916 13570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:06.576059 13570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:06.577255 13570 hybrid_clock.cc:648] HybridClock initialized: now 1790698866577239 us; error 30 us; skew 500 ppm
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:06.580168 13570 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:21:06.581095 13570 webserver.cc:460] Webserver started at http://127.4.200.126:46707/ using document root <none> and password file <none>
I20260929 16:21:06.581669 13570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:06.581789 13570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:06.585888 13570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:21:06.588152 13585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.589355 13570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:21:06.589505 13570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ce94c6ebfcd549ffb057d98f094a92be"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "1315d8fefa1f54a4939d49ebcf65cb40"
server_key_iv: "a1ce6fa6e48195d1e76966b17349ad93"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.590068 13570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:06.604768 13570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:06.608251 13570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:06.608464 13570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:06.616019 13570 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36037
I20260929 16:21:06.616045 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36037 every 8 connection(s)
I20260929 16:21:06.617058 13570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:06.621088 13638 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:21:06.622745 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Bootstrap starting.
I20260929 16:21:06.626067 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13570
I20260929 16:21:06.626566 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:45035
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:36037
--webserver_port=46849
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:21:06.639773 13638 log.cc:824] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:06.646628 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:21:06.647085 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Bootstrap complete.
I20260929 16:21:06.651414 13638 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:06.651806 13638 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce94c6ebfcd549ffb057d98f094a92be, State: Initialized, Role: FOLLOWER
I20260929 16:21:06.652339 13638 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:06.652453 13638 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:06.652539 13638 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:06.652631 13638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:21:06.654525 13638 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:06.654848 13638 leader_election.cc:302] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce94c6ebfcd549ffb057d98f094a92be; no voters:
I20260929 16:21:06.655189 13638 leader_election.cc:288] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:21:06.655270 13642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:21:06.655450 13642 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [term 2 LEADER]: Becoming Leader. State: Replica: ce94c6ebfcd549ffb057d98f094a92be, State: Running, Role: LEADER
I20260929 16:21:06.655740 13642 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } }
I20260929 16:21:06.656399 13638 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:06.656642 13642 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce94c6ebfcd549ffb057d98f094a92be. Latest consensus state: current_term: 2 leader_uuid: "ce94c6ebfcd549ffb057d98f094a92be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } } }
I20260929 16:21:06.656754 13642 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:06.657087 13646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:06.657220 13643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ce94c6ebfcd549ffb057d98f094a92be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94c6ebfcd549ffb057d98f094a92be" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36037 } } }
I20260929 16:21:06.657397 13643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:06.660212 13646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:06.660967 13646 catalog_manager.cc:1259] Loaded cluster ID: 918e2f31b1994703aabe7ea16cbc479d
I20260929 16:21:06.661023 13646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:06.663277 13646 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:06.663916 13646 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce94c6ebfcd549ffb057d98f094a92be: Loaded TSK: 0
I20260929 16:21:06.665099 13646 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:21:06.739972 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:06.740272 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:06.740351 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:06.744545 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:06.744645 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:06.744746 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:06.750217 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.65.keytab
--principal=oryx/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=46849
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:06.751390 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:06.752498 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:06.763114 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.763132 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.763121 13667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.763585 13640 server_base.cc:1034] running on GCE node
I20260929 16:21:06.763996 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:06.764638 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:06.765833 13640 hybrid_clock.cc:648] HybridClock initialized: now 1790698866765815 us; error 36 us; skew 500 ppm
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:06.768806 13640 init.cc:377] Logged in from keytab as oryx/127.4.200.65@KRBTEST.COM (short username oryx)
I20260929 16:21:06.769771 13640 webserver.cc:460] Webserver started at http://127.4.200.65:46849/ using document root <none> and password file <none>
I20260929 16:21:06.770519 13640 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:06.770635 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:06.774857 13640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:21:06.777045 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.778347 13640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:06.778519 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "fe9193f9f0764369a49f30a6fc98e0ef"
format_stamp: "Formatted at 2026-09-29 16:21:05 on dist-test-slave-s9ct"
server_key: "84fedb8f237b272eb50fd9202ed24b6c"
server_key_iv: "e79c4fe9d38ff3d998c645fe9544c550"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.779093 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:06.803231 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:06.806067 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:06.806275 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:06.806908 13640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:06.807742 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:06.807813 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.807885 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:06.807929 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.817162 13640 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:45035
I20260929 16:21:06.817188 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:45035 every 8 connection(s)
I20260929 16:21:06.818158 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:06.823282 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13640
I20260929 16:21:06.823716 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:36053
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:36037
--webserver_port=41439
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:06.830965 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.820360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:40817)
0929 16:21:06.821061 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:06.821111 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:06.821895 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:06.822897 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:06.822904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:06.823271 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:06.823813 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.823837 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.825135 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.825139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.825989 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.825998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.826836 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.826840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.827022 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.827629 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:06.827646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:06.830172 (+ 2526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":144,"threads_started":1}
W20260929 16:21:06.831434 13788 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:36037 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.200.126:36037: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
W20260929 16:21:06.831457 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:06.820231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40817 (local address 127.4.200.126:36037)
0929 16:21:06.820653 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:06.820668 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:06.821222 (+ 554us) server_negotiation.cc:407] Connection header received
0929 16:21:06.822231 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:06.822242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:06.822543 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:06.822809 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:06.824000 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.824989 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.826124 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.826662 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.826849 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.830525 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:40817: BlockingRecv error: recv got EOF from 127.4.200.65:40817 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:06.836869 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.831871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:52335)
0929 16:21:06.832128 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:06.832142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:06.832235 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:06.832459 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:06.832462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:06.832624 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:06.832719 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.832724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.833444 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.833450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.833994 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.834002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.834483 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.834487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.834579 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.835058 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:06.835074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:06.836780 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188}
W20260929 16:21:06.837280 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:06.831992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:52335 (local address 127.4.200.126:36037)
0929 16:21:06.832106 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:06.832110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:06.832208 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:21:06.832313 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:06.832316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:06.832363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:06.832448 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:06.832834 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.833338 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.834117 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.834341 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.834483 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.836933 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:52335: BlockingRecv error: recv got EOF from 127.4.200.65:52335 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:06.842350 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:06.837399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:42675)
0929 16:21:06.837484 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:06.837499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:06.837576 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:06.837860 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:06.837862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:06.837976 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:06.838039 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.838043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.839025 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.839028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.839523 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:06.839530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.840064 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.840066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:06.840121 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.840581 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:06.840593 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:06.842264 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 16:21:06.842566 13788 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:21:06.842730 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:06.837416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:42675 (local address 127.4.200.126:36037)
0929 16:21:06.837666 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:06.837670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:06.837684 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:21:06.837717 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:06.837720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:06.837758 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:06.837838 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:06.838139 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.838926 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.839633 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:06.839965 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:06.840059 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:06.842423 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:42675: BlockingRecv error: recv got EOF from 127.4.200.65:42675 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":116,"threads_started":1}
W20260929 16:21:06.935750 13792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:06.936003 13792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:06.936044 13792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:06.940055 13792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:06.940105 13792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:06.940166 13792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:06.945228 13792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.66.keytab
--principal=oryx/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=41439
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:06.946161 13792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:06.947185 13792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:06.956852 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.956957 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:06.956847 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:06.956960 13792 server_base.cc:1034] running on GCE node
I20260929 16:21:06.957504 13792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:06.958279 13792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:06.959455 13792 hybrid_clock.cc:648] HybridClock initialized: now 1790698866959435 us; error 33 us; skew 500 ppm
Sep 29 16:21:06 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:06.962216 13792 init.cc:377] Logged in from keytab as oryx/127.4.200.66@KRBTEST.COM (short username oryx)
I20260929 16:21:06.963122 13792 webserver.cc:460] Webserver started at http://127.4.200.66:41439/ using document root <none> and password file <none>
I20260929 16:21:06.963697 13792 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:06.963805 13792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:06.967628 13792 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 16:21:06.969623 13808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.970923 13792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:21:06.971051 13792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "39d68d414f334c64892bca8ba15e5fe0"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "453e377511d97873db9e506227d30a93"
server_key_iv: "42cd111c44ee995a098e6d003c3fed33"
server_key_version: "encryptionkey@0"
I20260929 16:21:06.971604 13792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:06.993063 13792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:06.996592 13792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:06.996799 13792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:06.997403 13792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:06.998220 13792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:06.998288 13792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:06.998356 13792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:06.998430 13792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:07.007110 13792 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:36053
I20260929 16:21:07.007143 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:36053 every 8 connection(s)
I20260929 16:21:07.008066 13792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:07.010183 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13792
I20260929 16:21:07.010691 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:40135
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:36037
--webserver_port=36253
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:21:07.020946 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.009796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:39591 (local address 127.4.200.126:36037)
0929 16:21:07.009983 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.009990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.010837 (+ 847us) server_negotiation.cc:407] Connection header received
0929 16:21:07.012414 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.012417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.012467 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.012547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.014096 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.014800 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.016618 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.016818 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.016889 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.020809 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:39591: BlockingRecv error: recv got EOF from 127.4.200.66:39591 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:21:07.021548 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.010162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:39591)
0929 16:21:07.010692 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.010741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.012153 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.012746 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.012757 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.013272 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.013927 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.013947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.014922 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.014928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.016487 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.016502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.016964 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.016971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.017213 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.017828 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.017854 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.020696 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":176,"threads_started":1}
W20260929 16:21:07.022004 13922 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:36037 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.200.126:36037: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:07.027352 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.022444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:49269)
0929 16:21:07.022566 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.022578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.022669 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.022904 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.022907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.023066 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.023144 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.023149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.023927 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.023933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.024490 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.024498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.025005 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.025009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.025104 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.025554 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.025565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.027214 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:21:07.027455 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.022579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:49269 (local address 127.4.200.126:36037)
0929 16:21:07.022699 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.022705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.022732 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:21:07.022768 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.022772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.022811 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.022891 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.023253 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.023804 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.024609 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.024892 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.025000 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.027326 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:49269: BlockingRecv error: recv got EOF from 127.4.200.66:49269 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:07.032320 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.027882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:52093)
0929 16:21:07.027974 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.027988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.028056 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.028272 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.028275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.028412 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.028473 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.028478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.029153 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.029157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.029648 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.029655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.030089 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.030092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.030170 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.030610 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.030621 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.032208 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 16:21:07.032442 13790 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.027956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:52093 (local address 127.4.200.126:36037)
0929 16:21:07.028055 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.028060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.028073 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:21:07.028135 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.028138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.028171 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.028263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.028580 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.029042 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.029761 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.029984 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.030090 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.032320 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:52093: BlockingRecv error: recv got EOF from 127.4.200.66:52093 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 16:21:07.032552 13922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:21:07.125006 13925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:07.125308 13925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:07.125351 13925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:07.129581 13925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:07.129647 13925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:07.129707 13925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:07.135039 13925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/krb5kdc/oryx_127.4.200.67.keytab
--principal=oryx/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=36253
--tserver_master_addrs=127.4.200.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:07.136125 13925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:07.137221 13925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:07.146672 13930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:07.146670 13933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:07.146651 13931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:07.146974 13925 server_base.cc:1034] running on GCE node
I20260929 16:21:07.147351 13925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:07.147949 13925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:07.149132 13925 hybrid_clock.cc:648] HybridClock initialized: now 1790698867149119 us; error 25 us; skew 500 ppm
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698867, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:07.151690 13925 init.cc:377] Logged in from keytab as oryx/127.4.200.67@KRBTEST.COM (short username oryx)
I20260929 16:21:07.152395 13925 webserver.cc:460] Webserver started at http://127.4.200.67:36253/ using document root <none> and password file <none>
I20260929 16:21:07.152889 13925 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:07.152951 13925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:07.156703 13925 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:07.159016 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:07.160247 13925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:07.160346 13925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "631b2d38556e432080a2b8fd746f4b0a"
format_stamp: "Formatted at 2026-09-29 16:21:06 on dist-test-slave-s9ct"
server_key: "cae0fa7b7a4dfe7b10d50a9cf2357deb"
server_key_iv: "6933fdf838cbb34a5f2117bcdb5abc1c"
server_key_version: "encryptionkey@0"
I20260929 16:21:07.160899 13925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:07.183179 13925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:07.186021 13925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:07.186167 13925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:07.186756 13925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:07.187505 13925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:07.187551 13925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:07.187590 13925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:07.187611 13925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:07.196337 13925 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:40135
I20260929 16:21:07.196358 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:40135 every 8 connection(s)
I20260929 16:21:07.197335 13925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:07.197669 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 13925
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:21:07.212213 13793 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.199621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47737 (local address 127.4.200.126:36037)
0929 16:21:07.199748 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.199758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.200625 (+ 867us) server_negotiation.cc:407] Connection header received
0929 16:21:07.201975 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.201978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.202053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.202144 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.203454 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.203971 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.205032 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.205248 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.205321 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.212030 (+ 6709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:47737: BlockingRecv error: recv got EOF from 127.4.200.67:47737 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:21:07.212756 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.199944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:47737)
0929 16:21:07.200450 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.200512 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.201765 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.202328 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.202334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.202864 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.203326 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.203338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.204076 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.204079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.204928 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.204936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.205366 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.205369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.205529 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.208400 (+ 2871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.208425 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.211855 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
W20260929 16:21:07.213200 14054 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:36037 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.200.126:36037: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
I20260929 16:21:07.214632 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.198458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.4.200.126:36037)
0929 16:21:07.198605 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.198609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.198821 (+ 212us) server_negotiation.cc:407] Connection header received
0929 16:21:07.199007 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.199009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.199056 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.199130 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.200027 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.200745 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.201603 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.201927 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.202054 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.209992 (+ 7938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:07.210000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:07.210020 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:07.210082 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:07.212340 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:07.212826 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:07.212832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:07.212838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:07.212989 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:07.213293 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:07.213297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:07.213299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:07.213763 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:07.213949 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:07.214261 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:21:07.214526 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:07.219347 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.213671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:41513)
0929 16:21:07.213925 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.213938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.214026 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.214247 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.214250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.214439 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.214564 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.214570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.215523 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.215525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.216334 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.216344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.216909 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.216913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.217004 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.217444 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.217458 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.219211 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260929 16:21:07.219435 13793 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.213783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41513 (local address 127.4.200.126:36037)
0929 16:21:07.213904 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.213908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.213993 (+ 85us) server_negotiation.cc:407] Connection header received
0929 16:21:07.214101 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.214104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.214141 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.214235 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.214670 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.215397 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.216464 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.216794 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.216908 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.219326 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:41513: BlockingRecv error: recv got EOF from 127.4.200.67:41513 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:07.224682 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.219855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:42849)
0929 16:21:07.220122 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.220137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.220225 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.220479 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.220481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.220653 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.220719 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.220723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.221437 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.221443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.221996 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.222004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.222519 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.222523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.222606 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.222983 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.222993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.224593 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205}
W20260929 16:21:07.224841 13793 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.219976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:42849 (local address 127.4.200.126:36037)
0929 16:21:07.220114 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.220118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.220184 (+ 66us) server_negotiation.cc:407] Connection header received
0929 16:21:07.220319 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.220333 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.220373 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.220465 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.220829 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.221330 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.222127 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.222359 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.222512 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.224716 (+ 2204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:42849: BlockingRecv error: recv got EOF from 127.4.200.67:42849 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:21:07.224910 14054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:07 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:07.848321 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:07.843158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:54635)
0929 16:21:07.843449 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:07.843464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:07.843577 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:07.843807 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:07.843808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:07.843980 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:07.844092 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.844097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.844817 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.844820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.845380 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:07.845387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.845797 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.845799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:07.845866 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.846307 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:07.846318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:07.848205 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260929 16:21:07.848500 14059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:07.843232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:54635 (local address 127.4.200.126:36037)
0929 16:21:07.843473 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:07.843478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:07.843523 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:21:07.843664 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:07.843667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:07.843720 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:07.843795 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:07.844186 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.844724 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.845479 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:07.845701 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:07.845776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:07.848330 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:54635: BlockingRecv error: recv got EOF from 127.4.200.65:54635 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:08.038367 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:08.033215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:60773)
0929 16:21:08.033511 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:08.033524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:08.033617 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:08.033904 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:08.033907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:08.034107 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:08.034204 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.034210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.035009 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.035011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.035612 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.035621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.036030 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.036032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.036092 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.036563 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:08.036575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:08.038249 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":131,"threads_started":1}
W20260929 16:21:08.038530 14059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:08.033344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:60773 (local address 127.4.200.126:36037)
0929 16:21:08.033476 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:08.033479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:08.033591 (+ 112us) server_negotiation.cc:407] Connection header received
0929 16:21:08.033705 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:08.033708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:08.033749 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:08.033821 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:08.034307 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.034893 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.035726 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.035937 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.036014 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.038377 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:60773: BlockingRecv error: recv got EOF from 127.4.200.66:60773 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:08.231568 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:08.225582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:34961)
0929 16:21:08.225925 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:08.225938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:08.226081 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:08.226378 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:08.226381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:08.226652 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:08.226748 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.226754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.227574 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.227580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.228207 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.228215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.228708 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.228712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.228804 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.229384 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:08.229407 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:08.231392 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":84,"threads_started":1}
W20260929 16:21:08.231632 14059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:08.225614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:34961 (local address 127.4.200.126:36037)
0929 16:21:08.225803 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:08.225810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:08.226065 (+ 255us) server_negotiation.cc:407] Connection header received
0929 16:21:08.226203 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:08.226207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:08.226265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:08.226358 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:08.226870 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.227455 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.228334 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.228588 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.228708 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.231520 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:34961: BlockingRecv error: recv got EOF from 127.4.200.67:34961 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:08 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:08.854475 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:08.849126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:47155)
0929 16:21:08.849459 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:08.849477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:08.849592 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:08.849903 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:08.849906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:08.850091 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:08.850197 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.850224 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.851007 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.851009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.851557 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:08.851563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.851956 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.851958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:08.852018 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.852496 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:08.852507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:08.854294 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":137,"threads_started":1}
W20260929 16:21:08.854656 14063 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:08.849231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:47155 (local address 127.4.200.126:36037)
0929 16:21:08.849467 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:08.849471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:08.849560 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:21:08.849697 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:08.849700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:08.849760 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:08.849850 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:08.850342 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.850913 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.851653 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:08.851865 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:08.851939 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:08.854493 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:47155: BlockingRecv error: recv got EOF from 127.4.200.65:47155 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:09.046373 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:09.039386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:44547)
0929 16:21:09.039644 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:09.039661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:09.039769 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:09.040162 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:09.040166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:09.040345 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:09.040469 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.040476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.041727 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.041730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.042281 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.042289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.042975 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.042978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.043054 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.043681 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:09.043698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:09.046180 (+ 2482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260929 16:21:09.046789 14063 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:09.039718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:44547 (local address 127.4.200.126:36037)
0929 16:21:09.039891 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:09.039897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:09.039916 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:21:09.039966 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:09.039970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:09.040032 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:09.040128 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:09.040607 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.041615 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.042434 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.042853 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.042973 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.046311 (+ 3338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:44547: BlockingRecv error: recv got EOF from 127.4.200.66:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:09.238086 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:09.232386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:41559)
0929 16:21:09.232698 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:09.232715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:09.232858 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:09.233135 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:09.233138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:09.233348 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:09.233438 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.233443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.234224 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.234229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.234904 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.234913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.235371 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.235374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.235457 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.236010 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:09.236022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:09.237949 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
W20260929 16:21:09.238220 14063 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:09.232559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41559 (local address 127.4.200.126:36037)
0929 16:21:09.232699 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:09.232705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:09.232795 (+ 90us) server_negotiation.cc:407] Connection header received
0929 16:21:09.232964 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:09.232968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:09.233021 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:09.233124 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:09.233554 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.234110 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.235026 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.235259 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.235370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.238090 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:41559: BlockingRecv error: recv got EOF from 127.4.200.67:41559 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:09 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:09.861506 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:09.855329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:56417)
0929 16:21:09.855667 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:09.855685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:09.855817 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:09.856168 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:09.856172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:09.856452 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:09.856609 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.856618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.857592 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.857595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.858219 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:09.858227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.858776 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.858779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:09.858848 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.859343 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:09.859355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:09.861316 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
W20260929 16:21:09.861564 14067 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:09.855411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56417 (local address 127.4.200.126:36037)
0929 16:21:09.855674 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:09.855678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:09.855767 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:21:09.855936 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:09.855942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:09.856018 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:09.856121 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:09.856817 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.857493 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.858370 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:09.858670 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:09.858757 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:09.861438 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:56417: BlockingRecv error: recv got EOF from 127.4.200.65:56417 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:10.052271 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:10.047224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:44415)
0929 16:21:10.047463 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:10.047476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:10.047570 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:10.047796 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:10.047799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:10.047966 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:10.048060 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.048065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.048774 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.048776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.049321 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.049328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.049749 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.049751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.049813 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.050363 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:10.050375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:10.052152 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
W20260929 16:21:10.052527 14067 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:10.047325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:44415 (local address 127.4.200.126:36037)
0929 16:21:10.047484 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:10.047487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:10.047523 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:21:10.047650 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:10.047652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:10.047695 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:10.047762 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:10.048153 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.048685 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.049416 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.049649 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.049726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.052290 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:44415: BlockingRecv error: recv got EOF from 127.4.200.66:44415 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:10.243999 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:10.238942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:38359)
0929 16:21:10.239247 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:10.239260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:10.239389 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:10.239795 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:10.239798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:10.239977 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:10.240073 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.240078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.240771 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.240773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.241302 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.241309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.241691 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.241694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.241752 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.242207 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:10.242218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:10.243893 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":122,"threads_started":1}
W20260929 16:21:10.244127 14067 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:10.238997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:38359 (local address 127.4.200.126:36037)
0929 16:21:10.239157 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:10.239161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:10.239362 (+ 201us) server_negotiation.cc:407] Connection header received
0929 16:21:10.239580 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:10.239581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:10.239654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:10.239749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:10.240172 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.240672 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.241396 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.241602 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.241674 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.244012 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:38359: BlockingRecv error: recv got EOF from 127.4.200.67:38359 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:10 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:10.867971 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:10.862345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:54403)
0929 16:21:10.862695 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:10.862721 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:10.862836 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:10.863101 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:10.863104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:10.863292 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:10.863393 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.863398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.864185 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.864187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.864883 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:10.864892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.865328 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.865331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:10.865390 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.865872 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:10.865884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:10.867819 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
W20260929 16:21:10.868129 14071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:10.862504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:54403 (local address 127.4.200.126:36037)
0929 16:21:10.862770 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:10.862773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:10.862786 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:21:10.862921 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:10.862922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:10.862963 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:10.863038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:10.863526 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.864084 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.864999 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:10.865226 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:10.865304 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:10.867985 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:54403: BlockingRecv error: recv got EOF from 127.4.200.65:54403 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:11.059000 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:11.053101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:39705)
0929 16:21:11.053442 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:11.053457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:11.053581 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:11.053859 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:11.053862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:11.054060 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:11.054173 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.054178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.054955 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.054958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.055501 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.055509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.055962 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.055965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.056038 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.056679 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:11.056699 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:11.058883 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
W20260929 16:21:11.059234 14071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:11.053203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:39705 (local address 127.4.200.126:36037)
0929 16:21:11.053359 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:11.053366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:11.053582 (+ 216us) server_negotiation.cc:407] Connection header received
0929 16:21:11.053669 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:11.053671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:11.053719 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:11.053797 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:11.054292 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.054852 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.055616 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.055833 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.055919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.059107 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:39705: BlockingRecv error: recv got EOF from 127.4.200.66:39705 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:11.250545 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:11.244816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:47681)
0929 16:21:11.245115 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:11.245128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:11.245238 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:11.245482 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:11.245484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:11.245657 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:11.245754 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.245760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.246576 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.246578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.247113 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.247121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.247596 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.247598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.247658 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.248229 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:11.248241 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:11.250335 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
W20260929 16:21:11.250837 14071 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:11.245000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47681 (local address 127.4.200.126:36037)
0929 16:21:11.245165 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:11.245168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:11.245183 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:11.245331 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:11.245333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:11.245378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:11.245450 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:11.245896 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.246459 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.247236 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.247479 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.247584 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.250610 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:47681: BlockingRecv error: recv got EOF from 127.4.200.67:47681 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:11 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:11.874811 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:11.868902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:46583)
0929 16:21:11.869187 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:11.869201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:11.869287 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:11.869518 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:11.869520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:11.869705 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:11.869801 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.869806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.870650 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.870653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.871256 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:11.871266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.871754 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.871757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:11.871844 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.872368 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:11.872379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:11.874649 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20260929 16:21:11.875041 14075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:11.869032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:46583 (local address 127.4.200.126:36037)
0929 16:21:11.869263 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:11.869266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:11.869279 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:21:11.869359 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:11.869361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:11.869401 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:11.869485 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:11.869911 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.870527 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.871391 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:11.871617 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:11.871708 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:11.874894 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:46583: BlockingRecv error: recv got EOF from 127.4.200.65:46583 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:12.065806 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:12.059899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:57681)
0929 16:21:12.060223 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:12.060237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:12.060327 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:12.060619 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:12.060621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:12.060804 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:12.060900 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.060905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.061683 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.061686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.062266 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.062274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.062812 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.062816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.062905 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.063514 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:12.063530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:12.065685 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
W20260929 16:21:12.065975 14075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:12.060145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:57681 (local address 127.4.200.126:36037)
0929 16:21:12.060306 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:12.060310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:12.060326 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:21:12.060416 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:12.060418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:12.060460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:12.060569 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:12.061020 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.061569 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.062415 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.062690 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.062783 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.065843 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:57681: BlockingRecv error: recv got EOF from 127.4.200.66:57681 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:12.262436 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:12.251490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:52363)
0929 16:21:12.251804 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:12.251821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:12.251926 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:12.252341 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:12.252349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:12.252587 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:12.252695 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.252701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.253439 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.253441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.253985 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.253993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.254442 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.254447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.254515 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.260440 (+ 5925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:12.260452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:12.262272 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
W20260929 16:21:12.262694 14075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:12.251568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:52363 (local address 127.4.200.126:36037)
0929 16:21:12.251723 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:12.251730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:12.252085 (+ 355us) server_negotiation.cc:407] Connection header received
0929 16:21:12.252144 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:12.252150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:12.252212 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:12.252285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:12.252817 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.253346 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.254083 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.254299 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.254372 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.262562 (+ 8190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:52363: BlockingRecv error: recv got EOF from 127.4.200.67:52363 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:12 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:12.880746 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:12.875645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:53583)
0929 16:21:12.875908 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:12.875921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:12.876024 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:12.876273 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:12.876275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:12.876488 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:12.876582 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.876587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.877298 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.877301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.877845 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:12.877851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.878242 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.878245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:12.878304 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.878844 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:12.878855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:12.880622 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
W20260929 16:21:12.880934 14079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:12.875705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:53583 (local address 127.4.200.126:36037)
0929 16:21:12.875971 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:12.875978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:12.875996 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:21:12.876112 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:12.876114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:12.876159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:12.876241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:12.876689 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.877210 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.877941 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:12.878153 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:12.878227 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:12.880764 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:53583: BlockingRecv error: recv got EOF from 127.4.200.65:53583 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":63,"threads_started":1}
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:13.072388 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:13.066612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:46585)
0929 16:21:13.066936 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:13.066954 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:13.067092 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:13.067362 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:13.067366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:13.067576 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:13.067669 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.067675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.068458 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.068463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.069015 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.069023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.069583 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.069586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.069689 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.070334 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:13.070353 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:13.072273 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
W20260929 16:21:13.072535 14079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:13.066730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:46585 (local address 127.4.200.126:36037)
0929 16:21:13.066903 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:13.066908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:13.067028 (+ 120us) server_negotiation.cc:407] Connection header received
0929 16:21:13.067200 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:13.067203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:13.067256 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:13.067346 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:13.067787 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.068336 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.069134 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.069426 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.069537 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.072408 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:46585: BlockingRecv error: recv got EOF from 127.4.200.66:46585 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:13.268563 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:13.263286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:40099)
0929 16:21:13.263575 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:13.263587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:13.263689 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:13.263915 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:13.263917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:13.264106 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:13.264197 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.264202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.264930 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.264932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.265518 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.265526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.265921 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.265923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.265978 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.266483 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:13.266494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:13.268380 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":74,"threads_started":1}
W20260929 16:21:13.268628 14079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:13.263332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:40099 (local address 127.4.200.126:36037)
0929 16:21:13.263454 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:13.263457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:13.263649 (+ 192us) server_negotiation.cc:407] Connection header received
0929 16:21:13.263753 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:13.263755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:13.263798 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:13.263865 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:13.264309 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.264840 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.265615 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.265828 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.265906 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.268511 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:40099: BlockingRecv error: recv got EOF from 127.4.200.67:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:13 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:13.887450 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:13.881546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:58169)
0929 16:21:13.881882 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:13.881904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:13.882006 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:13.882293 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:13.882296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:13.882504 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:13.882622 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.882628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.883426 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.883429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.884141 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:13.884152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.884651 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.884653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:13.884717 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.885247 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:13.885259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:13.887286 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
W20260929 16:21:13.887640 14083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:13.881658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:58169 (local address 127.4.200.126:36037)
0929 16:21:13.881890 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:13.881893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:13.881975 (+ 82us) server_negotiation.cc:407] Connection header received
0929 16:21:13.882111 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:13.882113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:13.882160 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:13.882245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:13.882740 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.883306 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.884285 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:13.884541 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:13.884638 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:13.887463 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:58169: BlockingRecv error: recv got EOF from 127.4.200.65:58169 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:14.079191 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:14.073221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:32775)
0929 16:21:14.073666 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:14.073681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:14.073784 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:14.074017 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:14.074019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:14.074215 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:14.074337 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.074343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.075255 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.075258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.076033 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.076044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.076483 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.076489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.076575 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.077062 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:14.077075 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:14.078993 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":113,"threads_started":1}
W20260929 16:21:14.079259 14083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:14.073393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:32775 (local address 127.4.200.126:36037)
0929 16:21:14.073597 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:14.073601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:14.073762 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:21:14.073859 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:14.073861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:14.073904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:14.073975 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:14.074561 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.075119 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.076153 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.076370 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.076450 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.079127 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:32775: BlockingRecv error: recv got EOF from 127.4.200.66:32775 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:14.275528 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:14.269395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:35327)
0929 16:21:14.269726 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:14.269740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:14.269848 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:14.270096 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:14.270098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:14.270274 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:14.270366 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.270371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.271226 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.271229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.271978 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.271990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.272572 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.272694 (+ 122us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.272776 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.273334 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:14.273346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:14.275390 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":139,"threads_started":1}
W20260929 16:21:14.275672 14083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:14.269507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:35327 (local address 127.4.200.126:36037)
0929 16:21:14.269709 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:14.269714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:14.269832 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:21:14.269932 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:14.269934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:14.269982 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:14.270055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:14.270514 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.271113 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.272071 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.272412 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.272531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.275527 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:35327: BlockingRecv error: recv got EOF from 127.4.200.67:35327 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:14 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:14.894327 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:14.888244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:40307)
0929 16:21:14.888551 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:14.888573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:14.888666 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:14.888936 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:14.888939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:14.889128 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:14.889231 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.889237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.890010 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.890013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.890752 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:14.890763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.891273 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.891276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:14.891409 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.892082 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:14.892093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:14.894178 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
W20260929 16:21:14.894659 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:14.888405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40307 (local address 127.4.200.126:36037)
0929 16:21:14.888641 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:14.888644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:14.888658 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:21:14.888748 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:14.888750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:14.888790 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:14.888862 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:14.889342 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.889866 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.890893 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:14.891145 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:14.891236 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:14.894402 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:40307: BlockingRecv error: recv got EOF from 127.4.200.65:40307 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:15.086503 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:15.080108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:50315)
0929 16:21:15.080409 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:15.080423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:15.080523 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:15.080785 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:15.080787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:15.080958 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:15.081067 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.081072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.082019 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.082022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.082658 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.082669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.083235 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.083238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.083331 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.083985 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:15.084003 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:15.086305 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
W20260929 16:21:15.086591 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:15.080212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:50315 (local address 127.4.200.126:36037)
0929 16:21:15.080376 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:15.080380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:15.080498 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:21:15.080615 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:15.080617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:15.080671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:15.080749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:15.081184 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.081875 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.082865 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.083107 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.083226 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.086472 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:50315: BlockingRecv error: recv got EOF from 127.4.200.66:50315 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:15.282054 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:15.276356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:35509)
0929 16:21:15.276695 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:15.276711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:15.276834 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:15.277145 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:15.277148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:15.277369 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:15.277505 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.277514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.278354 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.278357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.278941 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.278949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.279406 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.279409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.279492 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.279980 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:15.279992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:15.281884 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
W20260929 16:21:15.282121 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:15.276461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:35509 (local address 127.4.200.126:36037)
0929 16:21:15.276626 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:15.276631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:15.276892 (+ 261us) server_negotiation.cc:407] Connection header received
0929 16:21:15.276929 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:15.276931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:15.277008 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:15.277105 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:15.277634 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.278233 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.279065 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.279292 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.279403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.282013 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:35509: BlockingRecv error: recv got EOF from 127.4.200.67:35509 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:15 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:15.901103 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:15.895367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:37423)
0929 16:21:15.895674 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:15.895688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:15.895791 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:15.896128 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:15.896130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:15.896317 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:15.896419 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.896424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.897239 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.897241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.897824 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:15.897833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.898311 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.898313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:15.898377 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.898950 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:15.898962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:15.900915 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
W20260929 16:21:15.901211 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:15.895439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:37423 (local address 127.4.200.126:36037)
0929 16:21:15.895720 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:15.895723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:15.895742 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:21:15.895918 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:15.895921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:15.895969 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:15.896059 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:15.896564 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.897114 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.897959 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:15.898196 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:15.898285 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:15.901064 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:37423: BlockingRecv error: recv got EOF from 127.4.200.65:37423 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":63,"threads_started":1}
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:16.093552 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:16.087463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:41839)
0929 16:21:16.087838 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:16.087852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:16.087951 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:16.088257 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:16.088259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:16.088519 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:16.088619 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.088626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.089603 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.089609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.090219 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.090228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.090721 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.090724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.090789 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.091410 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:16.091423 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:16.093412 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":198,"threads_started":1}
W20260929 16:21:16.093750 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:16.087651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:41839 (local address 127.4.200.126:36037)
0929 16:21:16.087793 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:16.087797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:16.087904 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:21:16.088086 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:16.088089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:16.088143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:16.088230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:16.088714 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.089460 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.090350 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.090603 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.090690 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.093559 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:41839: BlockingRecv error: recv got EOF from 127.4.200.66:41839 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:16.288661 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:16.282972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:47939)
0929 16:21:16.283274 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:16.283292 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:16.283482 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:16.283800 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:16.283803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:16.284014 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:16.284125 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.284131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.284903 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.284908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.285460 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.285468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.285931 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.285934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.286020 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.286601 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:16.286613 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:16.288469 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
W20260929 16:21:16.288728 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:16.283110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47939 (local address 127.4.200.126:36037)
0929 16:21:16.283282 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:16.283287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:16.283430 (+ 143us) server_negotiation.cc:407] Connection header received
0929 16:21:16.283622 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:16.283624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:16.283672 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:16.283768 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:16.284261 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.284793 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.285604 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.285820 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.285929 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.288608 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:47939: BlockingRecv error: recv got EOF from 127.4.200.67:47939 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:16 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:16.907743 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:16.902183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:38843)
0929 16:21:16.902501 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:16.902521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:16.902665 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:16.903087 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:16.903090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:16.903315 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:16.903411 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.903417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.904212 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.904214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.904777 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:16.904785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.905186 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.905188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:16.905248 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.905745 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:16.905756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:16.907622 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
W20260929 16:21:16.907932 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:16.902313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:38843 (local address 127.4.200.126:36037)
0929 16:21:16.902570 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:16.902574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:16.902590 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:21:16.902872 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:16.902875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:16.902931 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:16.903036 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:16.903528 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.904107 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.904875 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:16.905089 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:16.905173 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:16.907761 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:38843: BlockingRecv error: recv got EOF from 127.4.200.65:38843 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:17.100277 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:17.094560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:50165)
0929 16:21:17.094849 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:17.094862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:17.094957 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:17.095201 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:17.095203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:17.095382 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:17.095476 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.095481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.096357 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.096362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.097031 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.097042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.097503 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.097505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.097567 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.098085 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:17.098098 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:17.100152 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
W20260929 16:21:17.100610 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:17.094557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:50165 (local address 127.4.200.126:36037)
0929 16:21:17.094730 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:17.094734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:17.094947 (+ 213us) server_negotiation.cc:407] Connection header received
0929 16:21:17.095046 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:17.095048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:17.095094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:17.095162 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:17.095628 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.096201 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.097160 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.097395 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.097484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.100343 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:50165: BlockingRecv error: recv got EOF from 127.4.200.66:50165 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:17.302371 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:17.289438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:55375)
0929 16:21:17.289778 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:17.289796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:17.289945 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:17.290282 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:17.290285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:17.290578 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:17.290679 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.290685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.291604 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.291610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.292186 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.292194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.292694 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.292698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.292791 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.300351 (+ 7560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:17.300365 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:17.302249 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
W20260929 16:21:17.302623 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:17.289633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:55375 (local address 127.4.200.126:36037)
0929 16:21:17.289780 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:17.289786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:17.289882 (+ 96us) server_negotiation.cc:407] Connection header received
0929 16:21:17.290102 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:17.290105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:17.290165 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:17.290259 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:17.290833 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.291489 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.292337 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.292574 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.292693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.302495 (+ 9802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:55375: BlockingRecv error: recv got EOF from 127.4.200.67:55375 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:17 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:17.914163 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:17.908609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:50865)
0929 16:21:17.908917 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:17.908931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:17.909025 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:17.909299 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:17.909301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:17.909483 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:17.909581 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.909586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.910331 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.910334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.910956 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:17.910965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.911417 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.911419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:17.911475 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.912034 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:17.912045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:17.913987 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":128,"threads_started":1}
W20260929 16:21:17.914254 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:17.908720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:50865 (local address 127.4.200.126:36037)
0929 16:21:17.908963 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:17.908967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:17.908981 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:21:17.909128 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:17.909130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:17.909174 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:17.909262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:17.909686 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.910235 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.911090 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:17.911317 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:17.911399 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:17.914119 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:50865: BlockingRecv error: recv got EOF from 127.4.200.65:50865 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:18.107071 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:18.101143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:36789)
0929 16:21:18.101472 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:18.101484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:18.101624 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:18.101895 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:18.101898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:18.102114 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:18.102204 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.102210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.103078 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.103083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.103656 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.103664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.104133 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.104136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.104215 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.104819 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:18.104840 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:18.106927 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
W20260929 16:21:18.107390 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:18.101196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:36789 (local address 127.4.200.126:36037)
0929 16:21:18.101356 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:18.101360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:18.101591 (+ 231us) server_negotiation.cc:407] Connection header received
0929 16:21:18.101720 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:18.101722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:18.101768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:18.101870 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:18.102328 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.102963 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.103780 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.104011 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.104131 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.107136 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:36789: BlockingRecv error: recv got EOF from 127.4.200.66:36789 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:18.308650 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:18.303133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:51365)
0929 16:21:18.303434 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:18.303448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:18.303581 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:18.303890 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:18.303893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:18.304110 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:18.304202 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.304208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.304988 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.304994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.305571 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.305579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.306081 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.306085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.306173 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.306709 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:18.306720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:18.308517 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
W20260929 16:21:18.308789 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:18.303194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:51365 (local address 127.4.200.126:36037)
0929 16:21:18.303368 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:18.303373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:18.303525 (+ 152us) server_negotiation.cc:407] Connection header received
0929 16:21:18.303712 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:18.303714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:18.303782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:18.303865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:18.304338 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.304873 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.305717 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.305965 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.306079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.308644 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:51365: BlockingRecv error: recv got EOF from 127.4.200.67:51365 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:18 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:18.929827 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:18.915150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:54991)
0929 16:21:18.915493 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:18.915510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:18.915642 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:18.915956 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:18.915963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:18.916216 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:18.916346 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.916352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.917072 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.917074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.917622 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:18.917629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.918041 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.918044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:18.918110 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.927779 (+ 9669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:18.927797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:18.929708 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
W20260929 16:21:18.930015 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:18.915204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:54991 (local address 127.4.200.126:36037)
0929 16:21:18.915468 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:18.915473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:18.915592 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:21:18.915753 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:18.915756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:18.915817 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:18.915909 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:18.916452 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.916984 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.917727 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:18.917937 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:18.918010 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:18.929870 (+ 11860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:54991: BlockingRecv error: recv got EOF from 127.4.200.65:54991 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:19.113442 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:19.107878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:42609)
0929 16:21:19.108208 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:19.108224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:19.108338 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:19.108642 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:19.108644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:19.108849 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:19.108939 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.108945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.109719 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.109724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.110277 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.110284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.110774 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.110778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.110859 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.111380 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:19.111395 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:19.113269 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":85,"threads_started":1}
W20260929 16:21:19.113510 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:19.108006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:42609 (local address 127.4.200.126:36037)
0929 16:21:19.108196 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:19.108201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:19.108281 (+ 80us) server_negotiation.cc:407] Connection header received
0929 16:21:19.108463 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:19.108466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:19.108509 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:19.108596 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:19.109062 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.109607 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.110417 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.110658 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.110771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.113394 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:42609: BlockingRecv error: recv got EOF from 127.4.200.66:42609 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:19.315287 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:19.309411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:39403)
0929 16:21:19.309770 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:19.309786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:19.309942 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:19.310226 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:19.310229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:19.310471 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:19.310560 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.310566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.311318 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.311324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.311876 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.311884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.312331 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.312335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.312420 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.312960 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:19.312974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:19.315116 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
W20260929 16:21:19.315351 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:19.309553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:39403 (local address 127.4.200.126:36037)
0929 16:21:19.309749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:19.309754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:19.309887 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:21:19.310052 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:19.310057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:19.310113 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:19.310203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:19.310674 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.311210 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.311991 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.312219 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.312328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.315235 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:39403: BlockingRecv error: recv got EOF from 127.4.200.67:39403 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:19 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:19.936436 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:19.930737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:37881)
0929 16:21:19.931096 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:19.931111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:19.931228 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:19.931500 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:19.931502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:19.931723 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:19.931827 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.931832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.932623 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.932645 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.933204 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:19.933213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.933858 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.933861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:19.933930 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.934468 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:19.934480 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:19.936318 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
W20260929 16:21:19.936853 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:19.930892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:37881 (local address 127.4.200.126:36037)
0929 16:21:19.931138 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:19.931141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:19.931174 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:21:19.931309 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:19.931311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:19.931354 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:19.931428 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:19.931969 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.932504 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.933472 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:19.933742 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:19.933835 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:19.936502 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:37881: BlockingRecv error: recv got EOF from 127.4.200.65:37881 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:20.120585 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:20.114494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:42965)
0929 16:21:20.114824 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:20.114841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:20.114998 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:20.115315 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:20.115317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:20.115593 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:20.115693 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.115699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.116577 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.116581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.117177 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.117186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.117882 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.117885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.117949 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.118608 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:20.118620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:20.120448 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
W20260929 16:21:20.120822 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:20.114660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:42965 (local address 127.4.200.126:36037)
0929 16:21:20.114828 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:20.114835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:20.114934 (+ 99us) server_negotiation.cc:407] Connection header received
0929 16:21:20.115124 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:20.115130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:20.115186 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:20.115272 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:20.115843 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.116395 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.117409 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.117641 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.117780 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.120697 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:42965: BlockingRecv error: recv got EOF from 127.4.200.66:42965 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:20.321478 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:20.316098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:54129)
0929 16:21:20.316384 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:20.316400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:20.316523 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:20.316862 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:20.316865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:20.317056 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:20.317159 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.317164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.317903 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.317907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.318584 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.318594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.319011 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.319014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.319075 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.319629 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:20.319640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:20.321356 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
W20260929 16:21:20.321620 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:20.316156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:54129 (local address 127.4.200.126:36037)
0929 16:21:20.316308 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:20.316312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:20.316524 (+ 212us) server_negotiation.cc:407] Connection header received
0929 16:21:20.316670 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:20.316673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:20.316728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:20.316807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:20.317261 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.317785 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.318691 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.318910 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.318991 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.321481 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:54129: BlockingRecv error: recv got EOF from 127.4.200.67:54129 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:20 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:20.943557 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:20.937494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:49455)
0929 16:21:20.937792 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:20.937814 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:20.937911 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:20.938181 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:20.938183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:20.938359 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:20.938482 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.938488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.939639 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.939643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.940357 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:20.940366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.940879 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.940882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:20.940944 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.941457 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:20.941468 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:20.943371 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20260929 16:21:20.943655 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:20.937549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:49455 (local address 127.4.200.126:36037)
0929 16:21:20.937896 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:20.937900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:20.937915 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:20.938011 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:20.938013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:20.938056 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:20.938146 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:20.938707 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.939503 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.940494 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:20.940760 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:20.940861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:20.943494 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:49455: BlockingRecv error: recv got EOF from 127.4.200.65:49455 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":80,"threads_started":1}
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:21.127301 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:21.121407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:50845)
0929 16:21:21.121722 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:21.121736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:21.121849 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:21.122094 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:21.122096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:21.122272 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:21.122366 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.122370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.123143 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.123145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.123729 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.123737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.124136 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.124138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.124197 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.124780 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:21.124792 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:21.127101 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
W20260929 16:21:21.127364 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:21.121537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:50845 (local address 127.4.200.126:36037)
0929 16:21:21.121708 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:21.121711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:21.121815 (+ 104us) server_negotiation.cc:407] Connection header received
0929 16:21:21.121935 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:21.121938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:21.121982 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:21.122059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:21.122509 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.123038 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.123831 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.124039 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.124116 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.127238 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:50845: BlockingRecv error: recv got EOF from 127.4.200.66:50845 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.67@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:21.328279 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:21.322489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.67:44353)
0929 16:21:21.322808 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:21.322823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:21.322977 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:21.323272 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:21.323275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:21.323477 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:21.323567 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.323572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.324381 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.324387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.324956 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.324964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.325497 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.325501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.325588 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.326151 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:21.326162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:21.328095 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
W20260929 16:21:21.328349 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:21.322581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:44353 (local address 127.4.200.126:36037)
0929 16:21:21.322778 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:21.322783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:21.322916 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:21:21.323090 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:21.323094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:21.323148 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:21.323236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:21.323690 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.324253 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.325084 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.325392 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.325485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.328226 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.67:44353: BlockingRecv error: recv got EOF from 127.4.200.67:44353 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:21 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.65@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:21.950325 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:21.944544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.65:56457)
0929 16:21:21.944876 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:21.944891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:21.944992 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:21.945237 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:21.945239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:21.945421 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:21.945516 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.945521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.946297 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.946299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.946972 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:21.946982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.947458 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.947461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:21.947530 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.948147 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:21.948159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:21.950164 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":134,"threads_started":1}
W20260929 16:21:21.950518 14115 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:21.944740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56457 (local address 127.4.200.126:36037)
0929 16:21:21.944953 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:21.944956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:21.944971 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:21:21.945081 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:21.945083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:21.945125 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:21.945196 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:21.945655 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.946191 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.947095 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:21.947321 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:21.947420 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:21.950351 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.65:56457: BlockingRecv error: recv got EOF from 127.4.200.65:56457 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
Sep 29 16:21:22 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:21:22 dist-test-slave-s9ct krb5kdc[13057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.200.66@KRBTEST.COM for oryx/127.4.200.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:21:22.137090 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.130721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36037 (local address 127.4.200.66:58525)
0929 16:21:22.131815 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:22.131834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:22.131954 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:22.132324 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:22.132342 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:22.132588 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:22.132713 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.132723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.133702 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.133705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.134265 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.134273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.134746 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.134748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.134809 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.135265 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:22.135275 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:22.136962 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.200.126:36037: Server oryx/127.4.200.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":79,"threads_started":1}
W20260929 16:21:22.137210 14115 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:22.131081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:58525 (local address 127.4.200.126:36037)
0929 16:21:22.131207 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:22.131212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:22.131907 (+ 695us) server_negotiation.cc:407] Connection header received
0929 16:21:22.132109 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:22.132112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:22.132182 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:22.132258 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:22.132818 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.133603 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.134383 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.134650 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.134732 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.137097 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.200.66:58525: BlockingRecv error: recv got EOF from 127.4.200.66:58525 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:21:22.200484 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13640
I20260929 16:21:22.206259 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13792
I20260929 16:21:22.212285 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13925
I20260929 16:21:22.217898 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 13570
2026-09-29T16:21:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17991 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:22Z Disabled control of system clock
I20260929 16:21:22.248903 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:38897
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:21:22.357892 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:22.358165 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:22.358201 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:22.362108 14120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:22.362162 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:22.362178 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:22.362195 14120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:22.362210 14120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:22.366953 14120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:38897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:22.367980 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:22.369309 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:22.377774 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:22.377839 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:22.377812 14120 server_base.cc:1034] running on GCE node
W20260929 16:21:22.378005 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:22.378614 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:22.379669 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:22.380864 14120 hybrid_clock.cc:648] HybridClock initialized: now 1790698882380833 us; error 42 us; skew 500 ppm
I20260929 16:21:22.392180 14120 webserver.cc:460] Webserver started at http://127.4.200.126:44719/ using document root <none> and password file <none>
I20260929 16:21:22.392807 14120 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:22.392912 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:22.393177 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:22.395457 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "8ecf386278584783ae6a89ac596d6a03"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "3470b9587a462c21b8814b353407d9cd"
server_key_iv: "1c7a361b473e39e85604e03fddf678db"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.396044 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "8ecf386278584783ae6a89ac596d6a03"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "3470b9587a462c21b8814b353407d9cd"
server_key_iv: "1c7a361b473e39e85604e03fddf678db"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.399788 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:21:22.402256 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.403537 14120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:22.403662 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "8ecf386278584783ae6a89ac596d6a03"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "3470b9587a462c21b8814b353407d9cd"
server_key_iv: "1c7a361b473e39e85604e03fddf678db"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.403748 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:22.427619 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:22.428239 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:22.428386 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:22.435909 14120 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:38897
I20260929 16:21:22.435906 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:38897 every 8 connection(s)
I20260929 16:21:22.436934 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:22.440359 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:22.444998 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14120
I20260929 16:21:22.445151 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:21:22.445446 4897 external_mini_cluster.cc:1428] Setting key 1e5a9372506c060b92ab611f1e2df3e7
I20260929 16:21:22.446311 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03: Bootstrap starting.
I20260929 16:21:22.448961 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:22.449880 14187 log.cc:824] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:22.451982 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03: No bootstrap required, opened a new log
I20260929 16:21:22.456300 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.446941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.4.200.126:38897)
0929 16:21:22.447664 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:22.447682 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:22.447726 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:21:22.448516 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:22.448553 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:22.449061 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:22.449449 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:22.449949 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.451364 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.452397 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.453058 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.453213 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.454072 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:22.454078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:22.454096 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:22.454622 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:22.454688 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:22.454712 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:22.454812 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:22.454969 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 16:21:22.455240 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
I20260929 16:21:22.456655 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } }
I20260929 16:21:22.456893 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:22.456948 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecf386278584783ae6a89ac596d6a03, State: Initialized, Role: FOLLOWER
I20260929 16:21:22.457343 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } }
I20260929 16:21:22.457489 14187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:22.457576 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:22.457688 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:22.458642 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } }
I20260929 16:21:22.458957 14187 leader_election.cc:302] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ecf386278584783ae6a89ac596d6a03; no voters:
I20260929 16:21:22.459205 14187 leader_election.cc:288] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:21:22.459348 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:22.459591 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [term 1 LEADER]: Becoming Leader. State: Replica: 8ecf386278584783ae6a89ac596d6a03, State: Running, Role: LEADER
I20260929 16:21:22.459976 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } }
I20260929 16:21:22.460433 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:22.461546 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ecf386278584783ae6a89ac596d6a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } } }
I20260929 16:21:22.461678 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:22.462080 14201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:22.462039 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ecf386278584783ae6a89ac596d6a03. Latest consensus state: current_term: 1 leader_uuid: "8ecf386278584783ae6a89ac596d6a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf386278584783ae6a89ac596d6a03" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38897 } } }
I20260929 16:21:22.462188 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:22.465051 14201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:22.470010 14201 catalog_manager.cc:1347] Generated new cluster ID: 64a5b1e727e54127ab234b5da14f7f21
I20260929 16:21:22.470074 14201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:22.491106 14201 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:21:22.492362 14201 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:22.499697 14201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ecf386278584783ae6a89ac596d6a03: Generated new TSK 0
I20260929 16:21:22.500589 14201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:21:22.513706 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--builtin_ntp_servers=127.4.200.84:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:21:22.657624 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:22.657888 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:22.657927 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:22.662173 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:22.662230 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:22.662313 14211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:22.667073 14211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:22.668162 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:22.669566 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:22.679072 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:22.679072 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:22.679126 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:22.679361 14211 server_base.cc:1034] running on GCE node
I20260929 16:21:22.679781 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:22.680352 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:22.681506 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790698882681492 us; error 26 us; skew 500 ppm
I20260929 16:21:22.683369 14211 webserver.cc:460] Webserver started at http://127.4.200.65:38821/ using document root <none> and password file <none>
I20260929 16:21:22.683928 14211 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:22.683988 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:22.684218 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:22.686237 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "76d2f436342047609d8b775e2447a8f3"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "f1b8b09de33969b8005007eb3db4d37d"
server_key_iv: "1dda54bf2fa03615f3a4e530ed05a245"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.686880 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "76d2f436342047609d8b775e2447a8f3"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "f1b8b09de33969b8005007eb3db4d37d"
server_key_iv: "1dda54bf2fa03615f3a4e530ed05a245"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.690346 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:21:22.692958 14225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.694769 14211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:21:22.694912 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "76d2f436342047609d8b775e2447a8f3"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "f1b8b09de33969b8005007eb3db4d37d"
server_key_iv: "1dda54bf2fa03615f3a4e530ed05a245"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.695040 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:22.708189 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:22.708992 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:22.709204 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:22.709795 14211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:22.710800 14211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:22.710878 14211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.710969 14211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:22.711020 14211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.719674 14211 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:46321
I20260929 16:21:22.719726 14338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:46321 every 8 connection(s)
I20260929 16:21:22.720588 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:22.721835 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14211
I20260929 16:21:22.721975 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:21:22.722339 4897 external_mini_cluster.cc:1428] Setting key db929ab7c91343922a7a2dc1179ef957
I20260929 16:21:22.724422 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--builtin_ntp_servers=127.4.200.84:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:21:22.730180 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.722497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:55647 (local address 127.4.200.126:38897)
0929 16:21:22.722670 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:22.722673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:22.723960 (+ 1287us) server_negotiation.cc:407] Connection header received
0929 16:21:22.725521 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:22.725525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:22.725577 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:22.725653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:22.726548 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.727072 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.728236 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.728469 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.728578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.729683 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:22.729687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:22.729689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:22.729703 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:22.729728 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:22.729735 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:22.729811 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:22.729972 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:21:22.730030 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:21:22.731060 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.722997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38897 (local address 127.4.200.65:55647)
0929 16:21:22.723808 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:22.723862 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:22.725303 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:22.725820 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:22.725827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:22.725907 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:22.726362 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.726373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.727194 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.727197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.728116 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.728125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.728621 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.728625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.728799 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.729414 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:22.729415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:22.729454 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:22.729517 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:22.729812 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:22.729823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:22.729831 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:21:22.729960 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:21:22.730225 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260929 16:21:22.732290 14339 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38897
I20260929 16:21:22.732527 14339 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:22.733024 14339 heartbeater.cc:511] Master 127.4.200.126:38897 requested a full tablet report, sending...
I20260929 16:21:22.734694 14151 ts_manager.cc:194] Registered new tserver with Master: 76d2f436342047609d8b775e2447a8f3 (127.4.200.65:46321)
I20260929 16:21:22.736243 14151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.65:55647
W20260929 16:21:22.837759 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:22.838047 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:22.838086 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:22.842037 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:22.842098 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:22.842152 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:22.846869 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:22.847944 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:22.849304 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:22.858983 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:22.859131 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:22.859015 14342 server_base.cc:1034] running on GCE node
W20260929 16:21:22.859020 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:22.859633 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:22.860368 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:22.861521 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790698882861486 us; error 54 us; skew 500 ppm
I20260929 16:21:22.863444 14342 webserver.cc:460] Webserver started at http://127.4.200.66:36139/ using document root <none> and password file <none>
I20260929 16:21:22.864035 14342 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:22.864100 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:22.864353 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:22.866506 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "2f6cac345855471695feac1f391e597b"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "a47d6991edc8cef9064327de0fa21975"
server_key_iv: "2c79e3173fbc7839e2f44b8d0a22332f"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.867072 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "2f6cac345855471695feac1f391e597b"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "a47d6991edc8cef9064327de0fa21975"
server_key_iv: "2c79e3173fbc7839e2f44b8d0a22332f"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.870674 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:21:22.873225 14356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.874567 14342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:22.874665 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "2f6cac345855471695feac1f391e597b"
format_stamp: "Formatted at 2026-09-29 16:21:22 on dist-test-slave-s9ct"
server_key: "a47d6991edc8cef9064327de0fa21975"
server_key_iv: "2c79e3173fbc7839e2f44b8d0a22332f"
server_key_version: "encryptionkey@0"
I20260929 16:21:22.874797 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:22.886512 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:22.887148 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:22.887336 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:22.887897 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:22.888813 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:22.888859 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.888897 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:22.888953 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:22.898278 14342 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:41805
I20260929 16:21:22.898308 14469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:41805 every 8 connection(s)
I20260929 16:21:22.899283 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:22.900712 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14342
I20260929 16:21:22.900832 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:21:22.901175 4897 external_mini_cluster.cc:1428] Setting key 8e5743bbc7e2e4d32c690df42588335f
I20260929 16:21:22.903188 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--builtin_ntp_servers=127.4.200.84:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:21:22.909132 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.901147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:34705 (local address 127.4.200.126:38897)
0929 16:21:22.901289 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:22.901296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:22.902593 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 16:21:22.904055 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:22.904058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:22.904118 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:22.904224 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:22.905125 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.905647 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.906916 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.907240 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.907369 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.908518 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:22.908528 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:22.908531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:22.908542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:22.908560 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:22.908567 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:22.908646 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:22.908816 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:21:22.908875 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:21:22.909916 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:22.901658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38897 (local address 127.4.200.66:34705)
0929 16:21:22.902466 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:22.902520 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:22.903855 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:22.904408 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:22.904415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:22.904497 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:22.904979 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.904991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.905775 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.905784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.906802 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:22.906811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:22.907385 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:22.907389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:22.907560 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:22.908260 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:22.908261 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:22.908300 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:22.908372 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:22.908667 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:22.908681 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:22.908686 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:21:22.908805 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:21:22.909076 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":173,"threads_started":1}
I20260929 16:21:22.911087 14470 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38897
I20260929 16:21:22.911307 14470 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:22.911810 14470 heartbeater.cc:511] Master 127.4.200.126:38897 requested a full tablet report, sending...
I20260929 16:21:22.912810 14151 ts_manager.cc:194] Registered new tserver with Master: 2f6cac345855471695feac1f391e597b (127.4.200.66:41805)
I20260929 16:21:22.913378 14151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.66:34705
W20260929 16:21:23.014057 14473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:23.014305 14473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:23.014340 14473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:23.018383 14473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:23.018438 14473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:23.018548 14473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:23.023015 14473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:23.023888 14473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:23.025183 14473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:23.034237 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.034343 14478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.034439 14479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.034587 14473 server_base.cc:1034] running on GCE node
I20260929 16:21:23.034973 14473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:23.035529 14473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:23.036706 14473 hybrid_clock.cc:648] HybridClock initialized: now 1790698883036694 us; error 25 us; skew 500 ppm
I20260929 16:21:23.038364 14473 webserver.cc:460] Webserver started at http://127.4.200.67:37683/ using document root <none> and password file <none>
I20260929 16:21:23.038921 14473 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:23.038977 14473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:23.039122 14473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:23.040927 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "5a2cd84b93b34877a7173a182ac02c8c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "cb7e8f539da4577ac1c88dfe1426a85e"
server_key_iv: "e1e33ee8884b2b6f0c053020ed49917b"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.041438 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "5a2cd84b93b34877a7173a182ac02c8c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "cb7e8f539da4577ac1c88dfe1426a85e"
server_key_iv: "e1e33ee8884b2b6f0c053020ed49917b"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.044879 14473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:21:23.047268 14487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.048439 14473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 16:21:23.048588 14473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "5a2cd84b93b34877a7173a182ac02c8c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "cb7e8f539da4577ac1c88dfe1426a85e"
server_key_iv: "e1e33ee8884b2b6f0c053020ed49917b"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.048697 14473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:23.067868 14473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:23.068722 14473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:23.068900 14473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:23.069525 14473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:23.070497 14473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:23.070571 14473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.070641 14473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:23.070673 14473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.079579 14473 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:34459
I20260929 16:21:23.079633 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:34459 every 8 connection(s)
I20260929 16:21:23.080482 14473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:23.089388 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14473
I20260929 16:21:23.089524 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:21:23.089709 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.082303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:56991 (local address 127.4.200.126:38897)
0929 16:21:23.082491 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.082495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.083531 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 16:21:23.084994 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.084997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.085056 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.085160 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.086022 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:23.086630 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:23.087744 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:23.087969 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:23.088096 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:23.089206 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:23.089210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:23.089212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:23.089224 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:23.089242 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:23.089250 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:23.089335 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:23.089502 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:21:23.089563 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:21:23.089869 4897 external_mini_cluster.cc:1428] Setting key e154a579b78e7d50ebe2a7d43e0c8274
I20260929 16:21:23.090679 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.082738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38897 (local address 127.4.200.67:56991)
0929 16:21:23.083382 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:23.083427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:23.084775 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:23.085324 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:23.085330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:23.085410 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:23.085876 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:23.085887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:23.086752 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:23.086758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:23.087617 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:23.087625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:23.088137 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:23.088140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:23.088299 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:23.088910 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:23.088912 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:23.088950 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:23.089009 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:23.089349 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:23.089358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:23.089370 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:21:23.089488 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:21:23.089780 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260929 16:21:23.091826 14601 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38897
I20260929 16:21:23.092074 14601 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:23.092571 14601 heartbeater.cc:511] Master 127.4.200.126:38897 requested a full tablet report, sending...
I20260929 16:21:23.094197 14151 ts_manager.cc:194] Registered new tserver with Master: 5a2cd84b93b34877a7173a182ac02c8c (127.4.200.67:34459)
I20260929 16:21:23.094833 14151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.67:56991
I20260929 16:21:23.103925 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:21:23.106868 14190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:23.105826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.4.200.126:38897)
0929 16:21:23.105929 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.105933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.106104 (+ 171us) server_negotiation.cc:407] Connection header received
0929 16:21:23.106206 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.106208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.106244 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.106311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.106723 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43244: BlockingRecv error: recv got EOF from 127.0.0.1:43244 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 16:21:23.107198 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14211
I20260929 16:21:23.113101 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14342
I20260929 16:21:23.118508 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14473
I20260929 16:21:23.123839 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14120
2026-09-29T16:21:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (903 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:23Z Disabled control of system clock
I20260929 16:21:23.151855 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33397
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:21:23.260488 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:23.260749 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:23.260787 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:23.265030 14612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:23.265105 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:23.265132 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:23.265149 14612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:23.265164 14612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:23.270267 14612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:23.271714 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:23.273368 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:23.282106 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.282102 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.282158 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.282439 14612 server_base.cc:1034] running on GCE node
I20260929 16:21:23.283015 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:23.284111 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:23.285306 14612 hybrid_clock.cc:648] HybridClock initialized: now 1790698883285291 us; error 29 us; skew 500 ppm
I20260929 16:21:23.287199 14612 webserver.cc:460] Webserver started at http://127.4.200.126:33237/ using document root <none> and password file <none>
I20260929 16:21:23.287776 14612 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:23.287869 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:23.288110 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:23.290112 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "160dd7bb2f140da14aead44d10c7ef04"
server_key_iv: "a914e8e7b2d26146fef474160c5b38ac"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.290729 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "160dd7bb2f140da14aead44d10c7ef04"
server_key_iv: "a914e8e7b2d26146fef474160c5b38ac"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.294925 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:21:23.297561 14626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.298933 14612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:23.299077 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "160dd7bb2f140da14aead44d10c7ef04"
server_key_iv: "a914e8e7b2d26146fef474160c5b38ac"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.299208 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:23.303833 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:23.304453 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:23.304653 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:23.311703 14612 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:33397
I20260929 16:21:23.311734 14678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:33397 every 8 connection(s)
I20260929 16:21:23.312634 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:23.315826 14679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:23.317093 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14612
I20260929 16:21:23.317232 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:21:23.317548 4897 external_mini_cluster.cc:1428] Setting key 3c27fd91053e278b60c0fe673aedc52e
I20260929 16:21:23.322189 14679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0: Bootstrap starting.
I20260929 16:21:23.323194 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.319014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.4.200.126:33397)
0929 16:21:23.319633 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.319648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.319683 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:21:23.320332 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.320350 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.320704 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.320977 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.321843 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:23.321848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:23.321860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:23.321871 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:23.321928 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:23.321945 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:23.322067 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:23.322194 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 16:21:23.322505 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260929 16:21:23.324667 14679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:23.325477 14679 log.cc:824] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:23.327471 14679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0: No bootstrap required, opened a new log
I20260929 16:21:23.331064 14679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } }
I20260929 16:21:23.331338 14679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:23.331400 14679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5c48b7da664ac8b7fb1a84ec0b7fb0, State: Initialized, Role: FOLLOWER
I20260929 16:21:23.331872 14679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } }
I20260929 16:21:23.331996 14679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:23.332034 14679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:23.332108 14679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:23.332957 14679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } }
I20260929 16:21:23.333240 14679 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5c48b7da664ac8b7fb1a84ec0b7fb0; no voters:
I20260929 16:21:23.333480 14679 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:21:23.333611 14684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:23.333870 14684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5c48b7da664ac8b7fb1a84ec0b7fb0, State: Running, Role: LEADER
I20260929 16:21:23.334177 14684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } }
I20260929 16:21:23.334729 14679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:23.336139 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } } }
I20260929 16:21:23.336287 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:23.336499 14686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5c48b7da664ac8b7fb1a84ec0b7fb0. Latest consensus state: current_term: 1 leader_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5c48b7da664ac8b7fb1a84ec0b7fb0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33397 } } }
I20260929 16:21:23.336593 14686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:23.337212 14693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:23.339872 14693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:23.345863 14693 catalog_manager.cc:1347] Generated new cluster ID: 009a7095db0c41e98329aa320a72806e
I20260929 16:21:23.345956 14693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:23.366870 14693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:21:23.367259 14693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:23.378608 14693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5c48b7da664ac8b7fb1a84ec0b7fb0: Generated new TSK 0
I20260929 16:21:23.379397 14693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:21:23.392184 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--builtin_ntp_servers=127.4.200.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:21:23.508513 14703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:23.508900 14703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:23.508975 14703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:23.513438 14703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:23.513516 14703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:23.513579 14703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:23.518955 14703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:23.520098 14703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:23.521505 14703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:23.531059 14709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.531035 14708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.531031 14711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.531503 14703 server_base.cc:1034] running on GCE node
I20260929 16:21:23.531890 14703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:23.532562 14703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:23.533735 14703 hybrid_clock.cc:648] HybridClock initialized: now 1790698883533718 us; error 33 us; skew 500 ppm
I20260929 16:21:23.535878 14703 webserver.cc:460] Webserver started at http://127.4.200.65:42593/ using document root <none> and password file <none>
I20260929 16:21:23.536460 14703 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:23.536523 14703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:23.536782 14703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:23.539072 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "ba7f88f2d2284085b05ebd55ea28b121"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "e66341289fb713c9472c883d7ae4a824"
server_key_iv: "873fcb1017630d055dc4c93cf55d6226"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.539642 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "ba7f88f2d2284085b05ebd55ea28b121"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "e66341289fb713c9472c883d7ae4a824"
server_key_iv: "873fcb1017630d055dc4c93cf55d6226"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.543382 14703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:21:23.545914 14717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.547277 14703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:23.547413 14703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "ba7f88f2d2284085b05ebd55ea28b121"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "e66341289fb713c9472c883d7ae4a824"
server_key_iv: "873fcb1017630d055dc4c93cf55d6226"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.547535 14703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:23.552215 14703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:23.552825 14703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:23.553009 14703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:23.553548 14703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:23.554497 14703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:23.554560 14703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.554625 14703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:23.554670 14703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.563405 14703 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:39171
I20260929 16:21:23.563426 14830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:39171 every 8 connection(s)
I20260929 16:21:23.564325 14703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:23.568647 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14703
I20260929 16:21:23.568771 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:21:23.569083 4897 external_mini_cluster.cc:1428] Setting key cc496b02b59d39e36d06a21750ce820e
I20260929 16:21:23.570551 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.566111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56317 (local address 127.4.200.126:33397)
0929 16:21:23.566258 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.566264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.567428 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 16:21:23.568690 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.568697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.568751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.568846 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.570038 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:23.570041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:23.570043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:23.570054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:23.570073 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:23.570081 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:23.570168 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:23.570327 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:21:23.570363 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:21:23.571372 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--builtin_ntp_servers=127.4.200.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:21:23.571625 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.566645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33397 (local address 127.4.200.65:56317)
0929 16:21:23.567283 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:23.567322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:23.568483 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:23.569030 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:23.569038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:23.569156 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:23.569755 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:23.569757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:23.569796 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:23.569895 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:23.570184 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:23.570193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:23.570199 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:21:23.570369 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:21:23.570637 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260929 16:21:23.572934 14831 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33397
I20260929 16:21:23.573177 14831 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:23.573570 14831 heartbeater.cc:511] Master 127.4.200.126:33397 requested a full tablet report, sending...
I20260929 16:21:23.575222 14643 ts_manager.cc:194] Registered new tserver with Master: ba7f88f2d2284085b05ebd55ea28b121 (127.4.200.65:39171)
W20260929 16:21:23.685490 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:23.685778 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:23.685822 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:23.689906 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:23.689971 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:23.690029 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:23.696002 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:23.697124 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:23.698557 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:21:23.708669 14834 server_base.cc:1034] running on GCE node
W20260929 16:21:23.708588 14839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.708643 14840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.708614 14842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.709375 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:23.710060 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:23.711253 14834 hybrid_clock.cc:648] HybridClock initialized: now 1790698883711191 us; error 74 us; skew 500 ppm
I20260929 16:21:23.713187 14834 webserver.cc:460] Webserver started at http://127.4.200.66:38241/ using document root <none> and password file <none>
I20260929 16:21:23.713815 14834 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:23.713913 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:23.714160 14834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:23.716352 14834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "7d081b0ac7464b0b91d4bc04b28a6ec1"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "618acb7d1b97b1547b8cbd561688239f"
server_key_iv: "ff2290a6708c9e81aa333cb0d3ede5db"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.716938 14834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "7d081b0ac7464b0b91d4bc04b28a6ec1"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "618acb7d1b97b1547b8cbd561688239f"
server_key_iv: "ff2290a6708c9e81aa333cb0d3ede5db"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.720854 14834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:21:23.723690 14848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.725087 14834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:21:23.725250 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "7d081b0ac7464b0b91d4bc04b28a6ec1"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "618acb7d1b97b1547b8cbd561688239f"
server_key_iv: "ff2290a6708c9e81aa333cb0d3ede5db"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.725389 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:23.730693 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:23.731356 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:23.731561 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:23.732137 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:23.733090 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:23.733166 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.733242 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:23.733299 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.743022 14834 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:37307
I20260929 16:21:23.743077 14961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:37307 every 8 connection(s)
I20260929 16:21:23.743993 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:23.748745 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14834
I20260929 16:21:23.748888 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:21:23.749212 4897 external_mini_cluster.cc:1428] Setting key 4ba0e15731bd9b7e51a6977c3ca209b5
I20260929 16:21:23.750278 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.745909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:60323 (local address 127.4.200.126:33397)
0929 16:21:23.746051 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.746055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.747180 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 16:21:23.748425 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.748428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.748503 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.748603 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.749824 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:23.749826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:23.749828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:23.749839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:23.749859 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:23.749868 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:23.749972 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:23.750109 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:21:23.750149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:21:23.751157 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.746315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33397 (local address 127.4.200.66:60323)
0929 16:21:23.747035 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:23.747075 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:23.748234 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:23.748775 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:23.748782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:23.748899 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:23.749551 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:23.749552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:23.749597 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:23.749662 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:23.749971 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:23.749978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:23.749982 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:21:23.750115 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:21:23.750343 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
I20260929 16:21:23.751204 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--builtin_ntp_servers=127.4.200.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:21:23.752275 14962 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33397
I20260929 16:21:23.752504 14962 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:23.752879 14962 heartbeater.cc:511] Master 127.4.200.126:33397 requested a full tablet report, sending...
I20260929 16:21:23.753839 14643 ts_manager.cc:194] Registered new tserver with Master: 7d081b0ac7464b0b91d4bc04b28a6ec1 (127.4.200.66:37307)
W20260929 16:21:23.865597 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:23.865911 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:23.865984 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:23.870030 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:23.870114 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:23.870208 14965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:23.874897 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:23.875972 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:23.877429 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:23.887164 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.887204 14965 server_base.cc:1034] running on GCE node
W20260929 16:21:23.887174 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:23.887164 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:23.887810 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:23.888506 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:23.889710 14965 hybrid_clock.cc:648] HybridClock initialized: now 1790698883889697 us; error 41 us; skew 500 ppm
I20260929 16:21:23.891606 14965 webserver.cc:460] Webserver started at http://127.4.200.67:35613/ using document root <none> and password file <none>
I20260929 16:21:23.892220 14965 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:23.892315 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:23.892583 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:23.895304 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "c3f27dc17667416fbdce9f6a5fa3b94c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "49e657c3172143d5e538cb3b33dda7ae"
server_key_iv: "c19741aafc09c1f5a620977715a1c46e"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.895919 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "c3f27dc17667416fbdce9f6a5fa3b94c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "49e657c3172143d5e538cb3b33dda7ae"
server_key_iv: "c19741aafc09c1f5a620977715a1c46e"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.899993 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:21:23.902767 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.904083 14965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:21:23.904219 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "c3f27dc17667416fbdce9f6a5fa3b94c"
format_stamp: "Formatted at 2026-09-29 16:21:23 on dist-test-slave-s9ct"
server_key: "49e657c3172143d5e538cb3b33dda7ae"
server_key_iv: "c19741aafc09c1f5a620977715a1c46e"
server_key_version: "encryptionkey@0"
I20260929 16:21:23.904350 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:23.909010 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:23.909628 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:23.909814 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:23.910344 14965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:23.911396 14965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:23.911465 14965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.911535 14965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:23.911584 14965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:23.920300 14965 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:44933
I20260929 16:21:23.920320 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:44933 every 8 connection(s)
I20260929 16:21:23.921231 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:23.927461 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.923212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:46045 (local address 127.4.200.126:33397)
0929 16:21:23.923368 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.923372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.924435 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 16:21:23.925721 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.925724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.925764 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.925843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.927028 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:23.927031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:23.927034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:21:23.927046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:23.927074 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:21:23.927083 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:23.927170 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:23.927314 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 16:21:23.927353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:21:23.927678 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 14965
I20260929 16:21:23.927807 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:21:23.928171 4897 external_mini_cluster.cc:1428] Setting key 63cc7de93d0b69ffcf12e11119f78d84
I20260929 16:21:23.928854 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:23.923597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33397 (local address 127.4.200.67:46045)
0929 16:21:23.924289 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:23.924330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:23.925505 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:23.926007 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:23.926014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:23.926121 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:23.926734 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:21:23.926736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:23.926780 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:21:23.926867 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:23.927175 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:23.927183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:23.927189 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:21:23.927406 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 16:21:23.927685 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260929 16:21:23.930095 15093 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33397
I20260929 16:21:23.930354 15093 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:23.930855 15093 heartbeater.cc:511] Master 127.4.200.126:33397 requested a full tablet report, sending...
I20260929 16:21:23.931989 14643 ts_manager.cc:194] Registered new tserver with Master: c3f27dc17667416fbdce9f6a5fa3b94c (127.4.200.67:44933)
I20260929 16:21:23.942708 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:21:23.945498 14682 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:21:23.944635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.4.200.126:33397)
0929 16:21:23.944782 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:23.944787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:23.944955 (+ 168us) server_negotiation.cc:407] Connection header received
0929 16:21:23.945047 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:23.945049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:23.945084 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:23.945147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:23.945369 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59902: BlockingRecv error: recv got EOF from 127.0.0.1:59902 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:21:23.945592 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14703
I20260929 16:21:23.951293 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14834
I20260929 16:21:23.957538 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14965
I20260929 16:21:23.965003 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 14612
2026-09-29T16:21:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (841 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:21:24 dist-test-slave-s9ct krb5kdc[15103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:21:24 dist-test-slave-s9ct krb5kdc[15103](info): set up 2 sockets
Sep 29 16:21:24 dist-test-slave-s9ct krb5kdc[15103](info): commencing operation
krb5kdc: starting...
W20260929 16:21:25.011374 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:21:25.134459 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36571
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:25.244671 15116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:25.244961 15116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:25.245033 15116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:25.249039 15116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:21:25.249130 15116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:25.249176 15116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:25.249235 15116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:21:25.249275 15116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:21:25.254446 15116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:25.255476 15116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:25.256513 15116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:25.264883 15122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:25.264910 15121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:25.264923 15116 server_base.cc:1034] running on GCE node
W20260929 16:21:25.264941 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:25.265652 15116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:25.266834 15116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:25.268007 15116 hybrid_clock.cc:648] HybridClock initialized: now 1790698885267979 us; error 42 us; skew 500 ppm
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:25.270888 15116 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:21:25.271734 15116 webserver.cc:460] Webserver started at http://127.4.200.126:36357/ using document root <none> and password file <none>
I20260929 16:21:25.272241 15116 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:25.272297 15116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:25.272459 15116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:25.274470 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "745d1bee8923459e807aad3d0e524e87"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "68012c01c3449dc31da14e8b6a13b952"
server_key_iv: "6e30ef384ec22007ae383e4a3229c5f0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.274984 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "745d1bee8923459e807aad3d0e524e87"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "68012c01c3449dc31da14e8b6a13b952"
server_key_iv: "6e30ef384ec22007ae383e4a3229c5f0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.278855 15116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:21:25.281698 15131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.282977 15116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:25.283146 15116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "745d1bee8923459e807aad3d0e524e87"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "68012c01c3449dc31da14e8b6a13b952"
server_key_iv: "6e30ef384ec22007ae383e4a3229c5f0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.283257 15116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:25.308552 15116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:25.311860 15116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:25.312062 15116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:25.318857 15116 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:36571
I20260929 16:21:25.319131 15183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:36571 every 8 connection(s)
I20260929 16:21:25.319828 15116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:21:25.320875 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 15116
I20260929 16:21:25.320996 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:21:25.321326 4897 external_mini_cluster.cc:1428] Setting key 422b062be96eb7e9378b64a140399378
I20260929 16:21:25.323081 15184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:25.328238 15184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: Bootstrap starting.
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:25.330529 15184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:25.331223 15184 log.cc:824] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:25.333091 15184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: No bootstrap required, opened a new log
I20260929 16:21:25.335754 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:25.322736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.4.200.126:36571)
0929 16:21:25.323256 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:25.323274 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:25.323313 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:21:25.323976 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:25.323992 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:25.324322 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:25.324679 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:25.325528 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.326614 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.327489 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.327998 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.328183 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:25.330977 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:25.330982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:25.330994 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:25.331024 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:25.332952 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.333471 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.333476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.333480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.333599 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.333883 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.333888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.333889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.334254 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:25.334499 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:25.334814 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:21:25.335082 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":118,"threads_started":1}
I20260929 16:21:25.336756 15184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } }
I20260929 16:21:25.337003 15184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:25.337085 15184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745d1bee8923459e807aad3d0e524e87, State: Initialized, Role: FOLLOWER
I20260929 16:21:25.337535 15184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } }
I20260929 16:21:25.337684 15184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:21:25.337769 15184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:21:25.337879 15184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:25.338788 15184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } }
I20260929 16:21:25.339121 15184 leader_election.cc:302] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745d1bee8923459e807aad3d0e524e87; no voters:
I20260929 16:21:25.339407 15184 leader_election.cc:288] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:21:25.339569 15189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:25.339877 15189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [term 1 LEADER]: Becoming Leader. State: Replica: 745d1bee8923459e807aad3d0e524e87, State: Running, Role: LEADER
I20260929 16:21:25.340157 15189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } }
I20260929 16:21:25.340587 15184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:21:25.341730 15191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745d1bee8923459e807aad3d0e524e87. Latest consensus state: current_term: 1 leader_uuid: "745d1bee8923459e807aad3d0e524e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } } }
I20260929 16:21:25.341856 15191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:25.342200 15198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:21:25.342144 15190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745d1bee8923459e807aad3d0e524e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745d1bee8923459e807aad3d0e524e87" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 36571 } } }
I20260929 16:21:25.342973 15190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87 [sys.catalog]: This master's current role is: LEADER
I20260929 16:21:25.345268 15198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:21:25.350531 15198 catalog_manager.cc:1347] Generated new cluster ID: 5d3c792667964b9b924f87a9b7ee1d9d
I20260929 16:21:25.350630 15198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:21:25.366032 15198 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:21:25.366963 15198 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:21:25.376745 15198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: Generated new TSK 0
I20260929 16:21:25.377450 15198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:21:25.410071 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:25.520123 15210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:25.520368 15210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:25.520411 15210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:25.524518 15210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:25.524578 15210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:25.524634 15210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:21:25.529827 15210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:25.530949 15210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:25.531968 15210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:21:25.541985 15210 server_base.cc:1034] running on GCE node
W20260929 16:21:25.541953 15216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:25.541962 15218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:25.542241 15215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:25.542706 15210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:25.543391 15210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:25.544589 15210 hybrid_clock.cc:648] HybridClock initialized: now 1790698885544561 us; error 39 us; skew 500 ppm
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:25.547526 15210 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:21:25.548375 15210 webserver.cc:460] Webserver started at http://127.4.200.65:38795/ using document root <none> and password file <none>
I20260929 16:21:25.548905 15210 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:25.548964 15210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:25.549132 15210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:25.551218 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "32c1811f232842b19bcf5511f9cd946e"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "7471954bbf38b8f82c8905fe65e7f376"
server_key_iv: "b5e73c8d9ca44e396b89257c7ad7790e"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.551729 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "32c1811f232842b19bcf5511f9cd946e"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "7471954bbf38b8f82c8905fe65e7f376"
server_key_iv: "b5e73c8d9ca44e396b89257c7ad7790e"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.555423 15210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:21:25.557916 15225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.559223 15210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:21:25.559345 15210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "32c1811f232842b19bcf5511f9cd946e"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "7471954bbf38b8f82c8905fe65e7f376"
server_key_iv: "b5e73c8d9ca44e396b89257c7ad7790e"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.559449 15210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:25.593990 15210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:25.598227 15210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:25.598501 15210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:25.599153 15210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:25.600010 15210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:25.600056 15210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.600126 15210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:25.600158 15210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.608990 15210 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:45371
I20260929 16:21:25.609012 15338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:45371 every 8 connection(s)
I20260929 16:21:25.609917 15210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:21:25.616912 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 15210
I20260929 16:21:25.617064 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:21:25.617413 4897 external_mini_cluster.cc:1428] Setting key 5e5bbf61951292d206a32fd44fcdd95c
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:25.625244 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:25.611699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:45813 (local address 127.4.200.126:36571)
0929 16:21:25.611855 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:25.611859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:25.612728 (+ 869us) server_negotiation.cc:407] Connection header received
0929 16:21:25.614090 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:25.614094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:25.614152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:25.614265 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:25.615556 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.616097 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.617369 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.617667 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.617784 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:25.620916 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:25.620921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:25.620924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:25.620960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:25.622631 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.623350 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.623354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.623357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.623457 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.623927 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.623931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.623932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.624120 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:25.624223 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:25.625007 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 16:21:25.625120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:21:25.626672 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:25.612077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36571 (local address 127.4.200.65:45813)
0929 16:21:25.612575 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:25.612618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:25.613872 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:25.614489 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:25.614495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:25.614849 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:25.615407 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:25.615421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.616229 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.616238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:25.617250 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:25.617259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.617825 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.617830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:25.618049 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:25.618811 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:25.618838 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:25.620713 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:25.622754 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:25.622764 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:25.622785 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:25.623209 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:25.623570 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:25.623577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:25.623582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:25.623801 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:25.624253 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:25.624263 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:25.624688 (+ 425us) client_negotiation.cc:769] Sending connection context
0929 16:21:25.624958 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 16:21:25.625387 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:21:25.627996 15339 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36571
I20260929 16:21:25.628232 15339 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:25.628775 15339 heartbeater.cc:511] Master 127.4.200.126:36571 requested a full tablet report, sending...
I20260929 16:21:25.630296 15148 ts_manager.cc:194] Registered new tserver with Master: 32c1811f232842b19bcf5511f9cd946e (127.4.200.65:45371)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:21:25.631875 15148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:45813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:21:25.648164 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:25.757997 15344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:25.758292 15344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:25.758373 15344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:25.762434 15344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:25.762511 15344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:25.762598 15344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:21:25.767851 15344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:25.768961 15344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:25.769999 15344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:25.779561 15350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:25.779570 15352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:25.779570 15349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:25.779582 15344 server_base.cc:1034] running on GCE node
I20260929 16:21:25.780220 15344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:25.780866 15344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:25.782034 15344 hybrid_clock.cc:648] HybridClock initialized: now 1790698885781995 us; error 57 us; skew 500 ppm
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:25.784901 15344 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:21:25.785772 15344 webserver.cc:460] Webserver started at http://127.4.200.66:34305/ using document root <none> and password file <none>
I20260929 16:21:25.786455 15344 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:25.786543 15344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:25.786800 15344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:25.788940 15344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "119830a31557461394697232f8e0346c"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "52e59f102cadcb4b339581cf643cf6b6"
server_key_iv: "8493c31732f450cbf02c8d8ea2d1c0c0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.789515 15344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "119830a31557461394697232f8e0346c"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "52e59f102cadcb4b339581cf643cf6b6"
server_key_iv: "8493c31732f450cbf02c8d8ea2d1c0c0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.793385 15344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:21:25.797039 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.798600 15344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:21:25.798724 15344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "119830a31557461394697232f8e0346c"
format_stamp: "Formatted at 2026-09-29 16:21:25 on dist-test-slave-s9ct"
server_key: "52e59f102cadcb4b339581cf643cf6b6"
server_key_iv: "8493c31732f450cbf02c8d8ea2d1c0c0"
server_key_version: "encryptionkey@0"
I20260929 16:21:25.798882 15344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:25.811659 15344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:25.815013 15344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:25.815212 15344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:25.815856 15344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:25.816707 15344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:25.816777 15344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.816851 15344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:25.816903 15344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:25.825717 15344 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38405
I20260929 16:21:25.825740 15472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38405 every 8 connection(s)
I20260929 16:21:25.826715 15344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:21:25.834621 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 15344
I20260929 16:21:25.834777 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:21:25.835165 4897 external_mini_cluster.cc:1428] Setting key 78cfb53a0687e16119bfabe54e16dc9c
Sep 29 16:21:25 dist-test-slave-s9ct krb5kdc[15103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:25.841632 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:25.828570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:46545 (local address 127.4.200.126:36571)
0929 16:21:25.828728 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:25.828732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:25.829576 (+ 844us) server_negotiation.cc:407] Connection header received
0929 16:21:25.831079 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:25.831083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:25.831141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:25.831251 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:25.832447 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.832981 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.834163 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.834379 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.834545 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:25.837450 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:25.837456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:25.837462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:25.837496 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:25.839195 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.839902 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.839909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.839914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.840008 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:25.840347 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:25.840353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:25.840357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:25.840632 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:25.840731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:25.841238 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 16:21:25.841385 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:21:25.842736 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:25.828961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36571 (local address 127.4.200.66:46545)
0929 16:21:25.829419 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:25.829462 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:25.830860 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:25.831433 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:25.831440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:25.831806 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:25.832293 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:25.832306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.833125 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.833128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:25.833991 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:25.833999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:25.834583 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:25.834588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:25.834798 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:25.835484 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:25.835501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:25.837275 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:25.839317 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:25.839324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:25.839343 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:25.839768 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:25.840113 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:25.840115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:25.840116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:25.840254 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:25.840744 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:25.840749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:25.840989 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 16:21:25.841148 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:21:25.841618 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260929 16:21:25.843978 15473 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36571
I20260929 16:21:25.844269 15473 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:25.844910 15473 heartbeater.cc:511] Master 127.4.200.126:36571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:21:25.846259 15148 ts_manager.cc:194] Registered new tserver with Master: 119830a31557461394697232f8e0346c (127.4.200.66:38405)
I20260929 16:21:25.846902 15148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:46545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:21:25.865866 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:21:25.978658 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:21:25.978927 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:21:25.978971 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:21:25.983086 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:21:25.983156 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:21:25.983219 15478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:21:25.988777 15478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:21:25.989804 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:21:25.990846 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:21:26.000865 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:26.000886 15478 server_base.cc:1034] running on GCE node
W20260929 16:21:26.000910 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:21:26.000874 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:21:26.001575 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:21:26.002267 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:21:26.003448 15478 hybrid_clock.cc:648] HybridClock initialized: now 1790698886003419 us; error 54 us; skew 500 ppm
Sep 29 16:21:26 dist-test-slave-s9ct krb5kdc[15103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698886, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:21:26.006240 15478 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:21:26.007220 15478 webserver.cc:460] Webserver started at http://127.4.200.67:42945/ using document root <none> and password file <none>
I20260929 16:21:26.007799 15478 fs_manager.cc:359] Metadata directory not provided
I20260929 16:21:26.007890 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:21:26.008126 15478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:21:26.010160 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "69dbab0219514a51863c0b51a3fe85f5"
format_stamp: "Formatted at 2026-09-29 16:21:26 on dist-test-slave-s9ct"
server_key: "91a3d083c3518c5dfd866d69589cd8bb"
server_key_iv: "b77099987beefac61252d49c0cfddaf5"
server_key_version: "encryptionkey@0"
I20260929 16:21:26.010754 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "69dbab0219514a51863c0b51a3fe85f5"
format_stamp: "Formatted at 2026-09-29 16:21:26 on dist-test-slave-s9ct"
server_key: "91a3d083c3518c5dfd866d69589cd8bb"
server_key_iv: "b77099987beefac61252d49c0cfddaf5"
server_key_version: "encryptionkey@0"
I20260929 16:21:26.014549 15478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:21:26.017295 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:26.018646 15478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:21:26.018779 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "69dbab0219514a51863c0b51a3fe85f5"
format_stamp: "Formatted at 2026-09-29 16:21:26 on dist-test-slave-s9ct"
server_key: "91a3d083c3518c5dfd866d69589cd8bb"
server_key_iv: "b77099987beefac61252d49c0cfddaf5"
server_key_version: "encryptionkey@0"
I20260929 16:21:26.018895 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:21:26.029273 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:21:26.032871 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:21:26.033092 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:21:26.033720 15478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:21:26.034655 15478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:21:26.034739 15478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:26.034811 15478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:21:26.034862 15478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:21:26.044647 15478 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:37015
I20260929 16:21:26.044687 15606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:37015 every 8 connection(s)
I20260929 16:21:26.045696 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:21:26.052484 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 15478
I20260929 16:21:26.052627 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:21:26.052942 4897 external_mini_cluster.cc:1428] Setting key bb89faa9e97ba677d7ac474372b6f291
Sep 29 16:21:26 dist-test-slave-s9ct krb5kdc[15103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698886, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:21:26.060159 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.047596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:36465 (local address 127.4.200.126:36571)
0929 16:21:26.047736 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.047739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.048497 (+ 758us) server_negotiation.cc:407] Connection header received
0929 16:21:26.049845 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.049848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.049907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.050012 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:26.051190 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.051724 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.053278 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.053597 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.053692 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.056543 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:26.056546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:26.056548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:26.056574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:26.058005 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:26.058634 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:26.058637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:26.058638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:26.058735 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:26.059115 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:26.059118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:26.059119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:26.059307 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:26.059384 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.059924 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.060035 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:21:26.060945 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.047906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:36571 (local address 127.4.200.67:36465)
0929 16:21:26.048342 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.048382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.049635 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.050187 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.050194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.050569 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:21:26.051028 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.051041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.051851 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.051854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.053102 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.053115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.053776 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.053780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.054003 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.054766 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:21:26.054786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:21:26.056375 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:21:26.058130 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:26.058135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:26.058150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:26.058520 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:26.058854 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:21:26.058857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:21:26.058859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:21:26.058998 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:21:26.059393 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:21:26.059399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:21:26.059638 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.059811 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.060086 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260929 16:21:26.062357 15607 heartbeater.cc:347] Connected to a master server at 127.4.200.126:36571
I20260929 16:21:26.062652 15607 heartbeater.cc:464] Registering TS with master...
I20260929 16:21:26.063227 15607 heartbeater.cc:511] Master 127.4.200.126:36571 requested a full tablet report, sending...
I20260929 16:21:26.064342 15148 ts_manager.cc:194] Registered new tserver with Master: 69dbab0219514a51863c0b51a3fe85f5 (127.4.200.67:37015)
I20260929 16:21:26.064922 15148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:36465
I20260929 16:21:26.067451 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:21:26.076838 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.069390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.4.200.126:36571)
0929 16:21:26.069527 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.069531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.069698 (+ 167us) server_negotiation.cc:407] Connection header received
0929 16:21:26.069787 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.069789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.069829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.069897 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:21:26.070622 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.071128 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.071977 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.072187 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.072266 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.073813 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:21:26.073817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:21:26.073819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:21:26.073840 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:21:26.075194 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:26.075603 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:26.075605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:26.075607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:26.075678 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:21:26.075995 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:21:26.075998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:21:26.075999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:21:26.076236 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:21:26.076347 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.076577 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.076717 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:21:26.079955 15148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:21:26.083171 15148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:21:26.098680 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.092837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.4.200.67:37015)
0929 16:21:26.093406 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.093419 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.093445 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:21:26.093529 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.093537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.093976 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.094208 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:26.094763 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.095588 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.097187 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.097923 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.098049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.098207 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.098412 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.098511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":146,"threads_started":1}
I20260929 16:21:26.098667 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.091942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37015 (local address 127.0.0.1:55074)
0929 16:21:26.092326 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.092374 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.092498 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.094449 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.094455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.094486 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:26.094607 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.094613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.095723 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.095728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.097047 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.097057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.098050 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.098053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.098167 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.098219 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.098356 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.098511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":20,"thread_start_us":126,"threads_started":1}
I20260929 16:21:26.098953 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.091987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:45371 (local address 127.0.0.1:59560)
0929 16:21:26.092913 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.092929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.093055 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.094705 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.094707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.094730 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:26.094810 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.094817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.096245 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.096248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.097479 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.097488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.098527 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.098531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.098640 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.098662 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.098715 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.098802 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":102,"thread_start_us":120,"threads_started":1}
I20260929 16:21:26.099469 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.092286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38405 (local address 127.0.0.1:52900)
0929 16:21:26.093484 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.093500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.093618 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.094547 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.094550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.094568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:26.094698 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.094704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.096268 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.096271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.097360 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.097369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.098527 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.098531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.099207 (+ 676us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.099231 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.099308 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.099377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":565,"thread_start_us":98,"threads_started":1}
I20260929 16:21:26.099589 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.093416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.4.200.66:38405)
0929 16:21:26.094014 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.094029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.094056 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:21:26.094157 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.094162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.094352 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.094840 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:26.094997 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.096116 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.097486 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.098336 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.098576 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.098724 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.099358 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.099454 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":146,"threads_started":1}
I20260929 16:21:26.100222 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.092821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.4.200.65:45371)
0929 16:21:26.094240 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.094255 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.094275 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:21:26.094341 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.094345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.094543 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.094699 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:26.094907 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.096086 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.097606 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.098367 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.099784 (+ 1417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.099933 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.099985 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.100080 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":88,"threads_started":1}
I20260929 16:21:26.101979 15541 tablet_service.cc:1467] Processing CreateTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d (DEFAULT_TABLE table=test-table [id=0f6f670dc7434a10ae20a9b7ce42104d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:26.102514 15407 tablet_service.cc:1467] Processing CreateTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d (DEFAULT_TABLE table=test-table [id=0f6f670dc7434a10ae20a9b7ce42104d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:26.103142 15541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee49ca6d41bc4577b06f63f3674cac7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:26.102908 15273 tablet_service.cc:1467] Processing CreateTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d (DEFAULT_TABLE table=test-table [id=0f6f670dc7434a10ae20a9b7ce42104d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:21:26.103562 15407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee49ca6d41bc4577b06f63f3674cac7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:26.103878 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee49ca6d41bc4577b06f63f3674cac7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:21:26.109155 15624 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Bootstrap starting.
I20260929 16:21:26.110985 15625 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Bootstrap starting.
I20260929 16:21:26.112692 15624 tablet_bootstrap.cc:654] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:26.113371 15626 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: Bootstrap starting.
I20260929 16:21:26.114142 15624 log.cc:824] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:26.114377 15625 tablet_bootstrap.cc:654] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:26.115201 15625 log.cc:824] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:26.115680 15626 tablet_bootstrap.cc:654] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: Neither blocks nor log segments found. Creating new log.
I20260929 16:21:26.116503 15624 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: No bootstrap required, opened a new log
I20260929 16:21:26.116714 15624 ts_tablet_manager.cc:1397] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:21:26.116899 15626 log.cc:824] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:21:26.117384 15625 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: No bootstrap required, opened a new log
I20260929 16:21:26.117630 15625 ts_tablet_manager.cc:1397] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260929 16:21:26.118948 15626 tablet_bootstrap.cc:492] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: No bootstrap required, opened a new log
I20260929 16:21:26.119159 15626 ts_tablet_manager.cc:1397] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 16:21:26.120741 15624 raft_consensus.cc:348] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.121049 15624 raft_consensus.cc:374] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:26.121137 15624 raft_consensus.cc:729] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dbab0219514a51863c0b51a3fe85f5, State: Initialized, Role: FOLLOWER
I20260929 16:21:26.121522 15625 raft_consensus.cc:348] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.121642 15624 consensus_queue.cc:260] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.121775 15625 raft_consensus.cc:374] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:26.121855 15625 raft_consensus.cc:729] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119830a31557461394697232f8e0346c, State: Initialized, Role: FOLLOWER
I20260929 16:21:26.122205 15626 raft_consensus.cc:348] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.122326 15625 consensus_queue.cc:260] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.122462 15607 heartbeater.cc:503] Master 127.4.200.126:36571 was elected leader, sending a full tablet report...
I20260929 16:21:26.122568 15626 raft_consensus.cc:374] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:21:26.122673 15626 raft_consensus.cc:729] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c1811f232842b19bcf5511f9cd946e, State: Initialized, Role: FOLLOWER
I20260929 16:21:26.122951 15624 ts_tablet_manager.cc:1428] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:21:26.123149 15626 consensus_queue.cc:260] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.123342 15625 ts_tablet_manager.cc:1428] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:21:26.123525 15473 heartbeater.cc:503] Master 127.4.200.126:36571 was elected leader, sending a full tablet report...
I20260929 16:21:26.124022 15339 heartbeater.cc:503] Master 127.4.200.126:36571 was elected leader, sending a full tablet report...
I20260929 16:21:26.125629 15626 ts_tablet_manager.cc:1428] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:21:26.183667 15632 raft_consensus.cc:482] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:21:26.183940 15632 raft_consensus.cc:504] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.185088 15632 leader_election.cc:288] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69dbab0219514a51863c0b51a3fe85f5 (127.4.200.67:37015), 119830a31557461394697232f8e0346c (127.4.200.66:38405)
I20260929 16:21:26.189302 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.185498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:44335 (local address 127.4.200.67:37015)
0929 16:21:26.185643 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.185648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.185877 (+ 229us) server_negotiation.cc:407] Connection header received
0929 16:21:26.186011 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.186014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.186071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.186148 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:26.186526 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.187082 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188303 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.188700 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188831 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.188885 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.189112 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.189178 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:21:26.189311 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.185578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40879 (local address 127.4.200.66:38405)
0929 16:21:26.185731 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.185735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.185938 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:21:26.186092 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.186095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.186178 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.186255 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:21:26.186645 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.187256 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188305 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.188710 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188835 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.188891 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.189124 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.189189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:21:26.189325 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.185362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:37015 (local address 127.4.200.65:44335)
0929 16:21:26.185778 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.185792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.185889 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.186222 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.186228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.186247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:26.186383 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.186426 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.187210 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.187216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.188177 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.188186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.188826 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.188965 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.189002 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.189090 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.189165 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":62,"thread_start_us":86,"threads_started":1}
I20260929 16:21:26.189332 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.185362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38405 (local address 127.4.200.65:40879)
0929 16:21:26.185804 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.185816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:21:26.185957 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:21:26.186319 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:21:26.186322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:21:26.186338 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:21:26.186523 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.186530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.187353 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.187359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.188179 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:21:26.188188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.188832 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.188835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:21:26.189001 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.189018 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:21:26.189102 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:21:26.189172 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":163,"thread_start_us":117,"threads_started":1}
I20260929 16:21:26.190166 15561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee49ca6d41bc4577b06f63f3674cac7d" candidate_uuid: "32c1811f232842b19bcf5511f9cd946e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dbab0219514a51863c0b51a3fe85f5" is_pre_election: true
I20260929 16:21:26.190235 15427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee49ca6d41bc4577b06f63f3674cac7d" candidate_uuid: "32c1811f232842b19bcf5511f9cd946e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119830a31557461394697232f8e0346c" is_pre_election: true
I20260929 16:21:26.190528 15427 raft_consensus.cc:2454] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c1811f232842b19bcf5511f9cd946e in term 0.
I20260929 16:21:26.190528 15561 raft_consensus.cc:2454] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c1811f232842b19bcf5511f9cd946e in term 0.
I20260929 16:21:26.190997 15228 leader_election.cc:302] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 119830a31557461394697232f8e0346c, 32c1811f232842b19bcf5511f9cd946e, 69dbab0219514a51863c0b51a3fe85f5; no voters:
I20260929 16:21:26.191288 15632 raft_consensus.cc:2781] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:21:26.191370 15632 raft_consensus.cc:482] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:21:26.191413 15632 raft_consensus.cc:3037] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:26.192435 15632 raft_consensus.cc:504] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.192823 15632 leader_election.cc:288] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [CANDIDATE]: Term 1 election: Requested vote from peers 69dbab0219514a51863c0b51a3fe85f5 (127.4.200.67:37015), 119830a31557461394697232f8e0346c (127.4.200.66:38405)
I20260929 16:21:26.193156 15561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee49ca6d41bc4577b06f63f3674cac7d" candidate_uuid: "32c1811f232842b19bcf5511f9cd946e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69dbab0219514a51863c0b51a3fe85f5"
I20260929 16:21:26.193302 15561 raft_consensus.cc:3037] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:26.194376 15561 raft_consensus.cc:2454] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c1811f232842b19bcf5511f9cd946e in term 1.
I20260929 16:21:26.194653 15427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee49ca6d41bc4577b06f63f3674cac7d" candidate_uuid: "32c1811f232842b19bcf5511f9cd946e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119830a31557461394697232f8e0346c"
I20260929 16:21:26.194736 15229 leader_election.cc:302] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c1811f232842b19bcf5511f9cd946e, 69dbab0219514a51863c0b51a3fe85f5; no voters:
I20260929 16:21:26.194794 15427 raft_consensus.cc:3037] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:21:26.194921 15632 raft_consensus.cc:2781] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:21:26.195197 15632 raft_consensus.cc:686] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 1 LEADER]: Becoming Leader. State: Replica: 32c1811f232842b19bcf5511f9cd946e, State: Running, Role: LEADER
I20260929 16:21:26.195546 15632 consensus_queue.cc:237] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } }
I20260929 16:21:26.196048 15427 raft_consensus.cc:2454] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c1811f232842b19bcf5511f9cd946e in term 1.
I20260929 16:21:26.199064 15145 catalog_manager.cc:5495] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c1811f232842b19bcf5511f9cd946e (127.4.200.65). New cstate: current_term: 1 leader_uuid: "32c1811f232842b19bcf5511f9cd946e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32c1811f232842b19bcf5511f9cd946e" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 45371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 } health_report { overall_health: UNKNOWN } } }
I20260929 16:21:26.226063 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.222431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.4.200.65:45371)
0929 16:21:26.222606 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:21:26.222611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:21:26.222631 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:21:26.222720 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:21:26.222722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:21:26.222778 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:21:26.222894 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:21:26.223162 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.223682 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.224574 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:21:26.224790 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:21:26.224896 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:21:26.225220 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:21:26.225655 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:21:26.225785 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:21:26.225852 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 16:21:26.225907 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:21:26.232316 15561 raft_consensus.cc:1261] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 1 FOLLOWER]: Refusing update from remote peer 32c1811f232842b19bcf5511f9cd946e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:26.232317 15427 raft_consensus.cc:1261] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 1 FOLLOWER]: Refusing update from remote peer 32c1811f232842b19bcf5511f9cd946e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:21:26.233072 15635 consensus_queue.cc:1035] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [LEADER]: Connected to new peer: Peer: permanent_uuid: "119830a31557461394697232f8e0346c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:26.233271 15632 consensus_queue.cc:1035] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [LEADER]: Connected to new peer: Peer: permanent_uuid: "69dbab0219514a51863c0b51a3fe85f5" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 37015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:21:26.239619 15638 mvcc.cc:205] Tried to move back new op lower bound from 7334702638000275456 to 7334702637859913728. Current Snapshot: MvccSnapshot[applied={T|T < 7334702638000275456}]
I20260929 16:21:26.241583 15640 mvcc.cc:205] Tried to move back new op lower bound from 7334702638000275456 to 7334702637859913728. Current Snapshot: MvccSnapshot[applied={T|T < 7334702638000275456}]
I20260929 16:21:26.243516 15639 mvcc.cc:205] Tried to move back new op lower bound from 7334702638000275456 to 7334702637859913728. Current Snapshot: MvccSnapshot[applied={T|T < 7334702638000275456}]
I20260929 16:21:26.249959 15145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:26.250181 15145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:21:26.252913 15145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745d1bee8923459e807aad3d0e524e87: Sending DeleteTablet for 3 replicas of tablet ee49ca6d41bc4577b06f63f3674cac7d
I20260929 16:21:26.253808 15273 tablet_service.cc:1514] Processing DeleteTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:26 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:59560
I20260929 16:21:26.253824 15541 tablet_service.cc:1514] Processing DeleteTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:26 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:55074
I20260929 16:21:26.254388 15648 tablet_replica.cc:330] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e: stopping tablet replica
I20260929 16:21:26.254426 15649 tablet_replica.cc:330] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: stopping tablet replica
I20260929 16:21:26.254439 15407 tablet_service.cc:1514] Processing DeleteTablet for tablet ee49ca6d41bc4577b06f63f3674cac7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:21:26 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52900
I20260929 16:21:26.254771 15649 raft_consensus.cc:2229] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:26.254771 15648 raft_consensus.cc:2229] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:21:26.255146 15649 raft_consensus.cc:2258] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:26.255172 15648 raft_consensus.cc:2258] T ee49ca6d41bc4577b06f63f3674cac7d P 32c1811f232842b19bcf5511f9cd946e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:26.255731 15650 tablet_replica.cc:330] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: stopping tablet replica
I20260929 16:21:26.255792 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 15210
I20260929 16:21:26.256063 15650 raft_consensus.cc:2229] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:21:26.256354 15650 raft_consensus.cc:2258] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:21:26.259177 15650 ts_tablet_manager.cc:1905] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:26.259574 15649 ts_tablet_manager.cc:1905] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:21:26.261602 15650 ts_tablet_manager.cc:1918] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:26.261736 15650 log.cc:1196] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/ee49ca6d41bc4577b06f63f3674cac7d
I20260929 16:21:26.262099 15650 ts_tablet_manager.cc:1939] T ee49ca6d41bc4577b06f63f3674cac7d P 119830a31557461394697232f8e0346c: Deleting consensus metadata
I20260929 16:21:26.263067 15649 ts_tablet_manager.cc:1918] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:21:26.263177 15649 log.cc:1196] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/ee49ca6d41bc4577b06f63f3674cac7d
I20260929 16:21:26.263479 15134 catalog_manager.cc:4841] TS 119830a31557461394697232f8e0346c (127.4.200.66:38405): tablet ee49ca6d41bc4577b06f63f3674cac7d (table test-table [id=0f6f670dc7434a10ae20a9b7ce42104d]) successfully deleted
I20260929 16:21:26.263584 15649 ts_tablet_manager.cc:1939] T ee49ca6d41bc4577b06f63f3674cac7d P 69dbab0219514a51863c0b51a3fe85f5: Deleting consensus metadata
W20260929 16:21:26.263900 15133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:45371 (error 108)
I20260929 16:21:26.264825 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 15344
I20260929 16:21:26.264863 15135 catalog_manager.cc:4841] TS 69dbab0219514a51863c0b51a3fe85f5 (127.4.200.67:37015): tablet ee49ca6d41bc4577b06f63f3674cac7d (table test-table [id=0f6f670dc7434a10ae20a9b7ce42104d]) successfully deleted
I20260929 16:21:26.265108 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:21:26.264715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:45371 (local address 127.0.0.1:59584)
0929 16:21:26.264912 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:21:26.264991 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 16:21:26.265709 15133 catalog_manager.cc:4584] TS 32c1811f232842b19bcf5511f9cd946e (127.4.200.65:45371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee49ca6d41bc4577b06f63f3674cac7d: Network error: Client connection negotiation failed: client connection to 127.4.200.65:45371: connect: Connection refused (error 111)
I20260929 16:21:26.270714 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 15478
I20260929 16:21:26.276050 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 15116
2026-09-29T16:21:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2313 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:21:26 dist-test-slave-s9ct krb5kdc[15654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:21:26 dist-test-slave-s9ct krb5kdc[15654](info): set up 2 sockets
Sep 29 16:21:26 dist-test-slave-s9ct krb5kdc[15654](info): commencing operation
krb5kdc: starting...
W20260929 16:21:27.315435 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:21:27 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:21:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangeradmin_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangeradmin_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangerlookup_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangerlookup_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
I20260929 16:21:27.489307 4897 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh_zHSI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/postgres -l logfile start
2026-09-29 16:21:30.185 UTC [15684] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:21:30.185 UTC [15684] LOG: listening on IPv4 address "127.4.200.84", port 50917
2026-09-29 16:21:30.188 UTC [15684] LOG: listening on Unix socket "/tmp/.s.PGSQL.50917"
2026-09-29 16:21:30.210 UTC [15687] LOG: database system was shut down at 2026-09-29 16:21:28 UTC
2026-09-29 16:21:30.214 UTC [15684] LOG: database system is ready to accept connections
I20260929 16:21:31.183208 4897 mini_postgres.cc:96] Postgres bound to 50917
2026-09-29 16:21:31.189 UTC [15695] FATAL: database "slave" does not exist
127.4.200.84:50917 - accepting connections
I20260929 16:21:31.190300 4897 mini_ranger.cc:161] Starting Ranger...
I20260929 16:21:31.207029 4897 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:21:31.816282 4897 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:21:31.816406 4897 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin
2026-09-29 16:21:32,099 [I] DB FLAVOR :POSTGRES
2026-09-29 16:21:32,099 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:21:32,099 [I] Checking connection..
2026-09-29 16:21:32,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:21:32,359 [I] Checking connection passed.
2026-09-29 16:21:32,360 [I] --------- Verifying version history table ---------
2026-09-29 16:21:32,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:21:32,616 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:21:32,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:21:32,868 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:21:32,868 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:21:32,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:21:33.129 UTC [15759] WARNING: there is no transaction in progress
2026-09-29 16:21:33,135 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:21:33,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:21:33,380 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:21:33,380 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:21:33,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:21:33,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:21:33,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s9ct.c.gcp-upstream.internal','N') ;"
2026-09-29 16:21:34,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:21:34,358 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:21:34,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:21:34,599 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:21:34,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:21:34,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:21:35,067 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:21:35,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:21:36.071 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.091 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.111 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.129 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.147 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.167 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.229 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.237 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.254 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.266 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.279 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.293 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.303 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.310 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.316 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:36.323 UTC [15867] WARNING: there is no transaction in progress
2026-09-29 16:21:37,050 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:21:37,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:21:37,307 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:21:37,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:21:37,542 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:21:37,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:21:37,794 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:21:37,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:21:38,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:21:38,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s9ct.c.gcp-upstream.internal';"
2026-09-29 16:21:38,486 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:21:38,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:21:38,726 [I] DB_PATCHES have already been applied
I20260929 16:21:38.731536 4897 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh_zHSI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:21:38.731600 4897 mini_ranger.cc:193] Using host: 127.4.200.84
I20260929 16:21:38.735110 4897 mini_ranger.cc:227] Ranger admin URL: http://127.4.200.84:40321
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:21:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:21:39.857735 4897 mini_ranger.cc:160] Time spent starting Ranger: real 8.667s user 0.005s sys 0.006s
Sep 29, 2026 4:21:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:21:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 4:21:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:21:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:22:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:22:00.723791 4897 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangerkms_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/rangerkms_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.200.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.200.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:22:00 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:22:00.828862 4897 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:22:00.839059 4897 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:22:01.790228 4897 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:22:01.790333 4897 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms
I20260929 16:22:01.790475 4897 mini_ranger_kms.cc:227] Using postgres at 127.4.200.84:50917
2026-09-29 16:22:01,995 [I] DB FLAVOR :POSTGRES
2026-09-29 16:22:01,995 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:22:01,995 [I] Checking connection
2026-09-29 16:22:01,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:22:02,230 [I] connection success
2026-09-29 16:22:02,230 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:22:02,230 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:22:02,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:22:02,470 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:22:02,471 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:22:02,471 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:22:02,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:22:02,750 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:22:03.212262 4897 mini_ranger_kms.cc:310] Created kms service
I20260929 16:22:03.430308 4897 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:22:03.475961 4897 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:22:03 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:03.916498 4897 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:22:03.916671 4897 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:22:03.916697 4897 mini_ranger_kms.cc:240] Using host: 127.4.200.84
I20260929 16:22:03.920423 4897 mini_ranger_kms.cc:276] Ranger KMS PID: 16047
I20260929 16:22:03.920578 4897 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.200.84:39875
Sep 29, 2026 4:22:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:22:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:22:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:22:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:22:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:22:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:22:06.208498 4897 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.380s user 0.018s sys 0.169s
I20260929 16:22:06.208630 4897 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:22:06.208741 4897 mini_ranger_kms.cc:401] 127.4.200.84:39875/kms/v1/keys
Sep 29 16:22:06 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790698926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:22:06 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
Sep 29 16:22:07 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
Sep 29 16:22:09 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:22:09.390952 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41033
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:41033
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:39875/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:09.533752 16098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:09.534097 16098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:09.534170 16098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:09.538233 16098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:22:09.538328 16098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:09.538365 16098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:09.538419 16098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:22:09.538455 16098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:22:09.543898 16098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.200.84:39875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:41033
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:09.544982 16098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:09.545994 16098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:09.555888 16103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:09.555897 16104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:09.556067 16098 server_base.cc:1034] running on GCE node
W20260929 16:22:09.556238 16106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:09.556852 16098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:09.558028 16098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:09.559259 16098 hybrid_clock.cc:648] HybridClock initialized: now 1790698929559242 us; error 34 us; skew 500 ppm
Sep 29 16:22:09 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698929, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:09.562115 16098 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:22:09.563174 16098 webserver.cc:460] Webserver started at http://127.4.200.126:44531/ using document root <none> and password file <none>
I20260929 16:22:09.563723 16098 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:09.563777 16098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:09.563999 16098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:09 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698929, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:09.611528 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "dc7cf7a9206745669d0306211e00eee0"
format_stamp: "Formatted at 2026-09-29 16:22:09 on dist-test-slave-s9ct"
server_key: "381e8c336e3e74e14ea00f2f756b73bc"
server_key_iv: "05e3d5ed8b459b0bde72a13523b7ff5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:09.612156 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "dc7cf7a9206745669d0306211e00eee0"
format_stamp: "Formatted at 2026-09-29 16:22:09 on dist-test-slave-s9ct"
server_key: "381e8c336e3e74e14ea00f2f756b73bc"
server_key_iv: "05e3d5ed8b459b0bde72a13523b7ff5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:09.616634 16098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 16:22:09.633874 16113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:09.635632 16098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:22:09.635783 16098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "dc7cf7a9206745669d0306211e00eee0"
format_stamp: "Formatted at 2026-09-29 16:22:09 on dist-test-slave-s9ct"
server_key: "381e8c336e3e74e14ea00f2f756b73bc"
server_key_iv: "05e3d5ed8b459b0bde72a13523b7ff5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:09.635929 16098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:09.669332 16098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:09.672209 16098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:09.672384 16098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:09.679448 16098 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:41033
I20260929 16:22:09.679492 16165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:41033 every 8 connection(s)
I20260929 16:22:09.680387 16098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:22:09.683195 16166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:22:09.688656 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: Bootstrap starting.
I20260929 16:22:09.690544 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 16098
I20260929 16:22:09.690699 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:22:09.691006 4897 external_mini_cluster.cc:1428] Setting key 1234a61944145ecb648a25055f415996
I20260929 16:22:09.691273 16166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:22:09.692096 16166 log.cc:824] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:22:09.694335 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: No bootstrap required, opened a new log
I20260929 16:22:09.698549 16166 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } }
I20260929 16:22:09.698843 16166 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:22:09.698916 16166 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7cf7a9206745669d0306211e00eee0, State: Initialized, Role: FOLLOWER
I20260929 16:22:09.699445 16166 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } }
I20260929 16:22:09.699649 16166 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:22:09.699754 16166 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:22:09.699891 16166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:22:09.700920 16166 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } }
Sep 29 16:22:09 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:09.701313 16166 leader_election.cc:302] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc7cf7a9206745669d0306211e00eee0; no voters:
I20260929 16:22:09.701730 16166 leader_election.cc:288] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:22:09.701829 16171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:22:09.702057 16171 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [term 1 LEADER]: Becoming Leader. State: Replica: dc7cf7a9206745669d0306211e00eee0, State: Running, Role: LEADER
I20260929 16:22:09.702482 16171 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } }
I20260929 16:22:09.703112 16166 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:22:09.704509 16173 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7cf7a9206745669d0306211e00eee0. Latest consensus state: current_term: 1 leader_uuid: "dc7cf7a9206745669d0306211e00eee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } } }
I20260929 16:22:09.704658 16173 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:09.704959 16172 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc7cf7a9206745669d0306211e00eee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cf7a9206745669d0306211e00eee0" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41033 } } }
I20260929 16:22:09.705259 16172 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:09.705497 16166 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:22:09.709956 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:09.692187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.4.200.126:41033)
0929 16:22:09.692731 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:09.692748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:09.692782 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:22:09.693669 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:09.693693 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:09.694099 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:09.694505 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:09.695298 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:09.696887 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:09.698013 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:09.698749 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:09.698942 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:09.702245 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:09.702251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:09.702270 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:09.702304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:09.704426 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:09.704981 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:09.704989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:09.704996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:09.705138 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:09.705446 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:09.705451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:09.705453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:09.705944 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:09.706171 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:09.706485 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:22:09.706785 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":158,"threads_started":1}
I20260929 16:22:09.710265 16182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:22:09.714600 16182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:22:09.722638 16182 catalog_manager.cc:1347] Generated new cluster ID: 4ee7310ee4f54dcda764cc6596458a80
I20260929 16:22:09.722719 16182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:22:09.730808 16182 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:22:09.732256 16182 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:22:09.749025 16182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: Generated new TSK 0
I20260929 16:22:09.750020 16182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:22:10.466763 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:39875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:10.610219 16211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:10.610623 16211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:10.610689 16211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:10.616580 16211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:10.616674 16211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:10.616775 16211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:22:10.623905 16211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.200.84:39875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:10.625345 16211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:10.626708 16211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:10.638623 16219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:10.638573 16218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:10.638604 16221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:10.639439 16211 server_base.cc:1034] running on GCE node
I20260929 16:22:10.639878 16211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:10.640599 16211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:10.641773 16211 hybrid_clock.cc:648] HybridClock initialized: now 1790698930641737 us; error 50 us; skew 500 ppm
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:10.644946 16211 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:22:10.645941 16211 webserver.cc:460] Webserver started at http://127.4.200.65:39207/ using document root <none> and password file <none>
I20260929 16:22:10.646646 16211 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:10.646739 16211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:10.646978 16211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:10.668779 16211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "51031fc1b7ef4c4ba8422bdae59728df"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "715576f6bfbc5ac1b25194b4f6543325"
server_key_iv: "ea754352c50fc65a02c233e7e081de66"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:10.669662 16211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "51031fc1b7ef4c4ba8422bdae59728df"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "715576f6bfbc5ac1b25194b4f6543325"
server_key_iv: "ea754352c50fc65a02c233e7e081de66"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:10.675102 16211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:22:10.702145 16228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:10.704499 16211 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260929 16:22:10.704687 16211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "51031fc1b7ef4c4ba8422bdae59728df"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "715576f6bfbc5ac1b25194b4f6543325"
server_key_iv: "ea754352c50fc65a02c233e7e081de66"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:10.704846 16211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:10.726791 16211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:10.730583 16211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:10.730836 16211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:10.731639 16211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:10.732729 16211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:10.732842 16211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:10.732966 16211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:10.733048 16211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:10.744421 16211 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:38847
I20260929 16:22:10.744864 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:38847 every 8 connection(s)
I20260929 16:22:10.745700 16211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:22:10.748569 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 16211
I20260929 16:22:10.748700 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:22:10.749006 4897 external_mini_cluster.cc:1428] Setting key 5b7f5cdc959670eb987bbe9edc7e190f
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:22:10.765152 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:10.748601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41033 (local address 127.4.200.65:39519)
0929 16:22:10.750303 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:10.750359 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:10.751685 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:10.752518 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:10.752529 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:10.752888 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:10.753560 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:10.753579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:10.754825 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:10.754829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:10.756208 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:10.756219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:10.756938 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:10.756942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:10.757142 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:10.757865 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:10.757886 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:10.759416 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:10.761692 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:10.761699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:10.761716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:10.762043 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:10.762363 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:10.762367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:10.762370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:10.762603 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:10.763069 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:10.763076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:10.763431 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 16:22:10.763631 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:22:10.763965 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:22:10.765221 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:10.748152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:39519 (local address 127.4.200.126:41033)
0929 16:22:10.750041 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:10.750049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:10.750477 (+ 428us) server_negotiation.cc:407] Connection header received
0929 16:22:10.751952 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:10.751955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:10.752013 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:10.752088 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:10.753737 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:10.754666 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:10.756364 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:10.756757 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:10.758620 (+ 1863us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:10.759595 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:10.759599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:10.759602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:10.759628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:10.761562 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:10.762170 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:10.762174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:10.762175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:10.762259 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:10.762730 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:10.762733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:10.762735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:10.762948 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:10.764786 (+ 1838us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:10.764906 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:22:10.765043 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1803,"thread_start_us":109,"threads_started":1}
I20260929 16:22:10.768214 16344 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41033
I20260929 16:22:10.768486 16344 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:10.769093 16344 heartbeater.cc:511] Master 127.4.200.126:41033 requested a full tablet report, sending...
I20260929 16:22:10.770834 16130 ts_manager.cc:194] Registered new tserver with Master: 51031fc1b7ef4c4ba8422bdae59728df (127.4.200.65:38847)
I20260929 16:22:10.772321 16130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:39519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:22:10.786535 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:39875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:22:10.924868 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:10.925284 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:10.925384 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:10.930236 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:10.930346 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:10.930483 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:22:10.936406 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.200.84:39875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:10.937661 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:10.938819 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:10.949168 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:10.949249 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:10.949168 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:10.950986 16350 server_base.cc:1034] running on GCE node
I20260929 16:22:10.951721 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:10.952590 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:10.953871 16350 hybrid_clock.cc:648] HybridClock initialized: now 1790698930953830 us; error 62 us; skew 500 ppm
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:10.957456 16350 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:22:10.958640 16350 webserver.cc:460] Webserver started at http://127.4.200.66:41463/ using document root <none> and password file <none>
I20260929 16:22:10.959249 16350 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:10.959347 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:10.959606 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:10 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:10.986032 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "d8f9a8e110b245069d82a383ff874a0e"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "3ca80a440ae6965721acb3143cee5dc0"
server_key_iv: "4bcfb28ba19cb8a388da59012d981fd6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:10.987963 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "d8f9a8e110b245069d82a383ff874a0e"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "3ca80a440ae6965721acb3143cee5dc0"
server_key_iv: "4bcfb28ba19cb8a388da59012d981fd6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:10.995625 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 16:22:11.024251 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.026206 16350 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260929 16:22:11.026453 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "d8f9a8e110b245069d82a383ff874a0e"
format_stamp: "Formatted at 2026-09-29 16:22:10 on dist-test-slave-s9ct"
server_key: "3ca80a440ae6965721acb3143cee5dc0"
server_key_iv: "4bcfb28ba19cb8a388da59012d981fd6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:11.026698 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:11.045557 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:11.049310 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:11.049559 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:11.050268 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:11.051347 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:11.051468 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.051577 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:11.051640 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.066201 16350 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38745
I20260929 16:22:11.067548 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:22:11.068681 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 16350
I20260929 16:22:11.068813 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:22:11.069134 4897 external_mini_cluster.cc:1428] Setting key 1682206e20ccbc7d0b86993e16c477ea
I20260929 16:22:11.072942 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38745 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:11.095468 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.077077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:56157 (local address 127.4.200.126:41033)
0929 16:22:11.077217 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.077224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.079102 (+ 1878us) server_negotiation.cc:407] Connection header received
0929 16:22:11.080454 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.080458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.080518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.080576 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:11.082354 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.083200 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.085352 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.085688 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.085766 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.091122 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:11.091127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:11.091130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:11.091159 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:11.092979 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.093623 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.093626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.093630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.093720 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.094129 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.094132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.094134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.094349 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:11.094491 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.095154 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.095287 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:22:11.096529 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.075954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41033 (local address 127.4.200.66:56157)
0929 16:22:11.078913 (+ 2959us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.078995 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.080084 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.081095 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.081106 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.081471 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:11.082173 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.082193 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.083361 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.083365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.085206 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.085217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.085974 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.085978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.086164 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.086846 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:11.086867 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:11.090902 (+ 4035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:11.093117 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:11.093124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:11.093141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:11.093506 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:11.093826 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:11.093829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:11.093831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:11.094028 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:11.094492 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:11.094499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:11.094863 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.095095 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.095439 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2719,"thread_start_us":154,"threads_started":1}
I20260929 16:22:11.097838 16480 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41033
I20260929 16:22:11.098106 16480 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:11.098724 16480 heartbeater.cc:511] Master 127.4.200.126:41033 requested a full tablet report, sending...
I20260929 16:22:11.100015 16130 ts_manager.cc:194] Registered new tserver with Master: d8f9a8e110b245069d82a383ff874a0e (127.4.200.66:38745)
I20260929 16:22:11.100750 16130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:56157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:22:11.117935 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:39875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:11.305708 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:11.306337 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:11.306507 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
W20260929 16:22:11.313925 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:11.314057 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:11.314182 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:22:11.322151 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.200.84:39875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:11.323926 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:11.325323 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:11.342612 16493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:11.344218 16485 server_base.cc:1034] running on GCE node
W20260929 16:22:11.344485 16491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:11.350111 16490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:11.350809 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:11.351742 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:11.352918 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790698931352844 us; error 86 us; skew 500 ppm
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698931, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:11.356735 16485 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:22:11.358057 16485 webserver.cc:460] Webserver started at http://127.4.200.67:43535/ using document root <none> and password file <none>
I20260929 16:22:11.358816 16485 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:11.358891 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:11.359146 16485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698931, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:11.381016 16485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "18d2e24a95da420fa3c45a0390754477"
format_stamp: "Formatted at 2026-09-29 16:22:11 on dist-test-slave-s9ct"
server_key: "b3ed0d6fdce6d990ab6c8d865aa8d704"
server_key_iv: "6a627c99c175cfe410f057fdd0b36bac"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:11.383538 16485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "18d2e24a95da420fa3c45a0390754477"
format_stamp: "Formatted at 2026-09-29 16:22:11 on dist-test-slave-s9ct"
server_key: "b3ed0d6fdce6d990ab6c8d865aa8d704"
server_key_iv: "6a627c99c175cfe410f057fdd0b36bac"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:11.390357 16485 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:22:11.419652 16500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.421665 16485 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260929 16:22:11.421842 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "18d2e24a95da420fa3c45a0390754477"
format_stamp: "Formatted at 2026-09-29 16:22:11 on dist-test-slave-s9ct"
server_key: "b3ed0d6fdce6d990ab6c8d865aa8d704"
server_key_iv: "6a627c99c175cfe410f057fdd0b36bac"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:11.421999 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:11.454319 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:11.458312 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:11.458631 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:11.459499 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:11.460641 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:11.460748 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.460843 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:11.460906 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:11.474495 16485 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:35287
I20260929 16:22:11.475673 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:22:11.480296 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 16485
I20260929 16:22:11.480425 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:22:11.480732 4897 external_mini_cluster.cc:1428] Setting key 99c72745f6ccf3ba8146a7ac7082fd2e
I20260929 16:22:11.480857 16613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:35287 every 8 connection(s)
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698931, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:11.499987 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.482940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:58833 (local address 127.4.200.126:41033)
0929 16:22:11.483059 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.483063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.486661 (+ 3598us) server_negotiation.cc:407] Connection header received
0929 16:22:11.487998 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.488002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.488059 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.488158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:11.489761 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.490685 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.492327 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.492676 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.492785 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.495714 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:11.495718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:11.495721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:11.495748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:11.497558 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.498165 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.498169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.498171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.498252 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.498681 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.498685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.498687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.498951 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:11.499055 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.499681 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.499812 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:22:11.501021 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.481827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41033 (local address 127.4.200.67:58833)
0929 16:22:11.486461 (+ 4634us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.486547 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.487719 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.488396 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.488408 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.488787 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:11.489525 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.489546 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.490827 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.490831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.492183 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.492194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.492848 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.492852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.493038 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.493760 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:11.493793 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:11.495538 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:11.497693 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:11.497699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:11.497717 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:11.498050 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:11.498354 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:11.498357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:11.498359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:11.498571 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:11.499062 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:11.499069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:11.499381 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.499590 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.499972 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4334,"thread_start_us":148,"threads_started":1}
I20260929 16:22:11.502315 16614 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41033
I20260929 16:22:11.502651 16614 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:11.503757 16614 heartbeater.cc:511] Master 127.4.200.126:41033 requested a full tablet report, sending...
I20260929 16:22:11.505041 16130 ts_manager.cc:194] Registered new tserver with Master: 18d2e24a95da420fa3c45a0390754477 (127.4.200.67:35287)
I20260929 16:22:11.506271 16130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:58833
I20260929 16:22:11.507892 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:22:11.520673 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.510324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.4.200.126:41033)
0929 16:22:11.510495 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.510500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.511003 (+ 503us) server_negotiation.cc:407] Connection header received
0929 16:22:11.511167 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.511187 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.511239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.511303 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:11.512235 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.513015 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.514724 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.515085 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.515166 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.516992 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:11.516996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:11.516999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:11.517025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:11.518896 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.519380 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.519384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.519386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.519469 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:11.519746 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:11.519749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:11.519751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:11.519961 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:11.520062 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.520401 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.520518 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:22:11.524582 16129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:22:11.528745 16129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:22:11.558091 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.543228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38745 (local address 127.0.0.1:55542)
0929 16:22:11.545892 (+ 2664us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.545903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.545968 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.548263 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.548269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.548291 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:22:11.548421 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.548427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.555006 (+ 6579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.555010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.556137 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.556149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.557494 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.557498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.557625 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.557666 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.557782 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.557931 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2609,"thread_start_us":90,"threads_started":1}
I20260929 16:22:11.561367 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.540693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:35287 (local address 127.0.0.1:47904)
0929 16:22:11.545500 (+ 4807us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.545559 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.545683 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.548904 (+ 3221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.548907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.548925 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:22:11.549013 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.549019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.552265 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.552269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.553607 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.553618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.560903 (+ 7285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.560906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.561016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.561057 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.561132 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.561213 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4619,"thread_start_us":107,"threads_started":1}
I20260929 16:22:11.562172 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.543602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.4.200.66:38745)
0929 16:22:11.546180 (+ 2578us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.546195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.546217 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:22:11.546291 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.546296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.547939 (+ 1643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.548096 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:22:11.549136 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.553811 (+ 4675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.556303 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.557174 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.557273 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.557434 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.561892 (+ 4458us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.562027 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474,"thread_start_us":58,"threads_started":1}
I20260929 16:22:11.562727 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.541796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.4.200.67:35287)
0929 16:22:11.546851 (+ 5055us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.546861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.546882 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:22:11.546951 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.546955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.548615 (+ 1660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.548756 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:22:11.551044 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.552129 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.558505 (+ 6376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.560564 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.560675 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.560825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.562469 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.562586 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4943,"thread_start_us":146,"threads_started":1}
I20260929 16:22:11.567638 16414 tablet_service.cc:1467] Processing CreateTablet for tablet dd37d084d5fa4b368d1789d096df3085 (DEFAULT_TABLE table=test-table [id=88035724e3294cb190fe5cd2149def3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:22:11.569000 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd37d084d5fa4b368d1789d096df3085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:22:11.580493 16548 tablet_service.cc:1467] Processing CreateTablet for tablet dd37d084d5fa4b368d1789d096df3085 (DEFAULT_TABLE table=test-table [id=88035724e3294cb190fe5cd2149def3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:22:11.583477 16548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd37d084d5fa4b368d1789d096df3085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:22:11.583760 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.544225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:38847 (local address 127.0.0.1:60234)
0929 16:22:11.546523 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.546535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.546610 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.577178 (+ 30568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.577182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.577211 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:22:11.577329 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.577337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.580832 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.580837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.582087 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.582104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.583319 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.583324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.583425 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.583457 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.583515 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.583599 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2244,"thread_start_us":90,"threads_started":1}
I20260929 16:22:11.584443 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.544399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.4.200.65:38847)
0929 16:22:11.563127 (+ 18728us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.563135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.576585 (+ 13450us) server_negotiation.cc:407] Connection header received
0929 16:22:11.576717 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.576723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.577001 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.577204 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:22:11.579477 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.580660 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.582268 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.583166 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.583983 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.584141 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.584198 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.584296 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18602,"thread_start_us":115,"threads_started":1}
I20260929 16:22:11.587635 16277 tablet_service.cc:1467] Processing CreateTablet for tablet dd37d084d5fa4b368d1789d096df3085 (DEFAULT_TABLE table=test-table [id=88035724e3294cb190fe5cd2149def3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:22:11.588840 16277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd37d084d5fa4b368d1789d096df3085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:22:11.593920 16634 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Bootstrap starting.
I20260929 16:22:11.597081 16634 tablet_bootstrap.cc:654] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Neither blocks nor log segments found. Creating new log.
I20260929 16:22:11.597386 16633 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Bootstrap starting.
I20260929 16:22:11.598100 16634 log.cc:824] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Log is configured to *not* fsync() on all Append() calls
I20260929 16:22:11.598929 16635 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: Bootstrap starting.
I20260929 16:22:11.599846 16633 tablet_bootstrap.cc:654] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Neither blocks nor log segments found. Creating new log.
I20260929 16:22:11.601181 16635 tablet_bootstrap.cc:654] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: Neither blocks nor log segments found. Creating new log.
I20260929 16:22:11.601720 16633 log.cc:824] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:22:11.603415 16634 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: No bootstrap required, opened a new log
I20260929 16:22:11.603710 16634 ts_tablet_manager.cc:1397] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 16:22:11.603802 16633 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: No bootstrap required, opened a new log
I20260929 16:22:11.603994 16633 ts_tablet_manager.cc:1397] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:22:11.608114 16633 raft_consensus.cc:348] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.608448 16633 raft_consensus.cc:374] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:22:11.608516 16633 raft_consensus.cc:729] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f9a8e110b245069d82a383ff874a0e, State: Initialized, Role: FOLLOWER
I20260929 16:22:11.609090 16633 consensus_queue.cc:260] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.609201 16634 raft_consensus.cc:348] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.609714 16634 raft_consensus.cc:374] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:22:11.609800 16634 raft_consensus.cc:729] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d2e24a95da420fa3c45a0390754477, State: Initialized, Role: FOLLOWER
I20260929 16:22:11.610335 16633 ts_tablet_manager.cc:1428] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:22:11.610874 16480 heartbeater.cc:503] Master 127.4.200.126:41033 was elected leader, sending a full tablet report...
I20260929 16:22:11.610347 16634 consensus_queue.cc:260] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.611975 16634 ts_tablet_manager.cc:1428] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:22:11.612465 16635 log.cc:824] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: Log is configured to *not* fsync() on all Append() calls
I20260929 16:22:11.613245 16614 heartbeater.cc:503] Master 127.4.200.126:41033 was elected leader, sending a full tablet report...
I20260929 16:22:11.619494 16635 tablet_bootstrap.cc:492] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: No bootstrap required, opened a new log
I20260929 16:22:11.619793 16635 ts_tablet_manager.cc:1397] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260929 16:22:11.625430 16635 raft_consensus.cc:348] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.625756 16635 raft_consensus.cc:374] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:22:11.625802 16635 raft_consensus.cc:729] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51031fc1b7ef4c4ba8422bdae59728df, State: Initialized, Role: FOLLOWER
I20260929 16:22:11.626338 16635 consensus_queue.cc:260] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.627480 16635 ts_tablet_manager.cc:1428] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:22:11.627916 16344 heartbeater.cc:503] Master 127.4.200.126:41033 was elected leader, sending a full tablet report...
I20260929 16:22:11.637396 16638 raft_consensus.cc:482] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:22:11.637627 16638 raft_consensus.cc:504] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.638934 16638 leader_election.cc:288] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d2e24a95da420fa3c45a0390754477 (127.4.200.67:35287), 51031fc1b7ef4c4ba8422bdae59728df (127.4.200.65:38847)
I20260929 16:22:11.644364 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.639748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:44277 (local address 127.4.200.65:38847)
0929 16:22:11.639869 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.639874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.640092 (+ 218us) server_negotiation.cc:407] Connection header received
0929 16:22:11.640260 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.640263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.640305 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.640350 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:22:11.640717 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.641511 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.643126 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.643704 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.643791 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.643854 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.644164 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.644236 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:22:11.645058 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.639389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:38847 (local address 127.4.200.66:44277)
0929 16:22:11.640016 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.640029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.640153 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.640473 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.640475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.640488 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:22:11.640610 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.640615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.641705 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.641708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.642993 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.643003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.643945 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.643948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.644040 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.644061 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.644824 (+ 763us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.644910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":104,"threads_started":1}
I20260929 16:22:11.645787 16294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd37d084d5fa4b368d1789d096df3085" candidate_uuid: "d8f9a8e110b245069d82a383ff874a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51031fc1b7ef4c4ba8422bdae59728df" is_pre_election: true
I20260929 16:22:11.646111 16294 raft_consensus.cc:2454] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f9a8e110b245069d82a383ff874a0e in term 0.
I20260929 16:22:11.646732 16370 leader_election.cc:302] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51031fc1b7ef4c4ba8422bdae59728df, d8f9a8e110b245069d82a383ff874a0e; no voters:
I20260929 16:22:11.647037 16638 raft_consensus.cc:2781] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:22:11.647106 16638 raft_consensus.cc:482] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:22:11.647143 16638 raft_consensus.cc:3037] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:22:11.648474 16638 raft_consensus.cc:504] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.648849 16638 leader_election.cc:288] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [CANDIDATE]: Term 1 election: Requested vote from peers 18d2e24a95da420fa3c45a0390754477 (127.4.200.67:35287), 51031fc1b7ef4c4ba8422bdae59728df (127.4.200.65:38847)
I20260929 16:22:11.649279 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.639558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:33237 (local address 127.4.200.67:35287)
0929 16:22:11.639696 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.639701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.639968 (+ 267us) server_negotiation.cc:407] Connection header received
0929 16:22:11.640189 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.640193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.640245 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.640329 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:22:11.645571 (+ 5242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.646333 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.647913 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.648534 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.648651 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.648714 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.649074 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.649159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:22:11.649577 16294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd37d084d5fa4b368d1789d096df3085" candidate_uuid: "d8f9a8e110b245069d82a383ff874a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51031fc1b7ef4c4ba8422bdae59728df"
I20260929 16:22:11.649699 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.639196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:35287 (local address 127.4.200.66:33237)
0929 16:22:11.639874 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.639889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:11.640053 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:11.645262 (+ 5209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:11.645265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:11.645288 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:22:11.645418 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.645424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.646544 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.646547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.647735 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:11.647749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.648829 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.648832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:11.648923 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.648946 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:22:11.649471 (+ 525us) client_negotiation.cc:240] Negotiation successful
0929 16:22:11.649548 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":131,"threads_started":1}
I20260929 16:22:11.649701 16294 raft_consensus.cc:3037] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:22:11.650511 16568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd37d084d5fa4b368d1789d096df3085" candidate_uuid: "d8f9a8e110b245069d82a383ff874a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d2e24a95da420fa3c45a0390754477" is_pre_election: true
I20260929 16:22:11.650835 16568 raft_consensus.cc:2454] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f9a8e110b245069d82a383ff874a0e in term 0.
I20260929 16:22:11.651202 16294 raft_consensus.cc:2454] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f9a8e110b245069d82a383ff874a0e in term 1.
I20260929 16:22:11.651175 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd37d084d5fa4b368d1789d096df3085" candidate_uuid: "d8f9a8e110b245069d82a383ff874a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d2e24a95da420fa3c45a0390754477"
I20260929 16:22:11.651288 16567 raft_consensus.cc:3037] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:22:11.651522 16370 leader_election.cc:302] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51031fc1b7ef4c4ba8422bdae59728df, d8f9a8e110b245069d82a383ff874a0e; no voters:
I20260929 16:22:11.651675 16638 raft_consensus.cc:2781] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:22:11.651929 16638 raft_consensus.cc:686] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 1 LEADER]: Becoming Leader. State: Replica: d8f9a8e110b245069d82a383ff874a0e, State: Running, Role: LEADER
I20260929 16:22:11.652268 16638 consensus_queue.cc:237] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } }
I20260929 16:22:11.652559 16567 raft_consensus.cc:2454] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f9a8e110b245069d82a383ff874a0e in term 1.
I20260929 16:22:11.655479 16129 catalog_manager.cc:5495] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f9a8e110b245069d82a383ff874a0e (127.4.200.66). New cstate: current_term: 1 leader_uuid: "d8f9a8e110b245069d82a383ff874a0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f9a8e110b245069d82a383ff874a0e" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 } health_report { overall_health: UNKNOWN } } }
I20260929 16:22:11.678293 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.673499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.4.200.66:38745)
0929 16:22:11.673621 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:11.673626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:11.673693 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:22:11.673905 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:11.673908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:11.673983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:11.674103 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:22:11.674507 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.675286 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.676183 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:11.676516 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:11.676622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:11.677010 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:22:11.677580 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:22:11.677706 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:11.678086 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 16:22:11.678148 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:22:11.690831 16294 raft_consensus.cc:1261] T dd37d084d5fa4b368d1789d096df3085 P 51031fc1b7ef4c4ba8422bdae59728df [term 1 FOLLOWER]: Refusing update from remote peer d8f9a8e110b245069d82a383ff874a0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:22:11.691691 16638 consensus_queue.cc:1035] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51031fc1b7ef4c4ba8422bdae59728df" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:22:11.693450 16567 raft_consensus.cc:1261] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 1 FOLLOWER]: Refusing update from remote peer d8f9a8e110b245069d82a383ff874a0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:22:11.694126 16638 consensus_queue.cc:1035] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d2e24a95da420fa3c45a0390754477" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 16:22:11 dist-test-slave-s9ct krb5kdc[15654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698930, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:11.702189 16648 mvcc.cc:205] Tried to move back new op lower bound from 7334702824196251648 to 7334702824050769920. Current Snapshot: MvccSnapshot[applied={T|T < 7334702824196251648}]
I20260929 16:22:11.705684 16649 mvcc.cc:205] Tried to move back new op lower bound from 7334702824196251648 to 7334702824050769920. Current Snapshot: MvccSnapshot[applied={T|T < 7334702824196251648}]
I20260929 16:22:11.708098 16651 mvcc.cc:205] Tried to move back new op lower bound from 7334702824196251648 to 7334702824050769920. Current Snapshot: MvccSnapshot[applied={T|T < 7334702824196251648}]
I20260929 16:22:11.731549 16129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:22:11.731760 16129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:22:11.734829 16129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc7cf7a9206745669d0306211e00eee0: Sending DeleteTablet for 3 replicas of tablet dd37d084d5fa4b368d1789d096df3085
I20260929 16:22:11.735705 16548 tablet_service.cc:1514] Processing DeleteTablet for tablet dd37d084d5fa4b368d1789d096df3085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:22:11 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:47904
I20260929 16:22:11.736279 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet dd37d084d5fa4b368d1789d096df3085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:22:11 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:55542
I20260929 16:22:11.736781 16658 tablet_replica.cc:330] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: stopping tablet replica
I20260929 16:22:11.736769 16277 tablet_service.cc:1514] Processing DeleteTablet for tablet dd37d084d5fa4b368d1789d096df3085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:22:11 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:60234
I20260929 16:22:11.737864 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 16211
I20260929 16:22:11.737900 16657 tablet_replica.cc:330] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: stopping tablet replica
I20260929 16:22:11.738229 16657 raft_consensus.cc:2229] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:22:11.738777 16657 raft_consensus.cc:2258] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:22:11.741984 16657 ts_tablet_manager.cc:1905] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:22:11.743072 16658 raft_consensus.cc:2229] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:22:11.744022 16658 raft_consensus.cc:2258] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:22:11.744979 16657 ts_tablet_manager.cc:1918] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:22:11.745118 16657 log.cc:1196] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/dd37d084d5fa4b368d1789d096df3085
I20260929 16:22:11.745512 16657 ts_tablet_manager.cc:1939] T dd37d084d5fa4b368d1789d096df3085 P 18d2e24a95da420fa3c45a0390754477: Deleting consensus metadata
I20260929 16:22:11.746928 16115 catalog_manager.cc:4841] TS 18d2e24a95da420fa3c45a0390754477 (127.4.200.67:35287): tablet dd37d084d5fa4b368d1789d096df3085 (table test-table [id=88035724e3294cb190fe5cd2149def3a]) successfully deleted
I20260929 16:22:11.747061 16658 ts_tablet_manager.cc:1905] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:22:11.750042 16658 ts_tablet_manager.cc:1918] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:22:11.750137 16658 log.cc:1196] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/dd37d084d5fa4b368d1789d096df3085
I20260929 16:22:11.750543 16658 ts_tablet_manager.cc:1939] T dd37d084d5fa4b368d1789d096df3085 P d8f9a8e110b245069d82a383ff874a0e: Deleting consensus metadata
I20260929 16:22:11.751921 16116 catalog_manager.cc:4841] TS d8f9a8e110b245069d82a383ff874a0e (127.4.200.66:38745): tablet dd37d084d5fa4b368d1789d096df3085 (table test-table [id=88035724e3294cb190fe5cd2149def3a]) successfully deleted
W20260929 16:22:11.756040 16117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:38847 (error 108)
I20260929 16:22:11.756820 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 16350
I20260929 16:22:11.759362 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:11.759014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:38847 (local address 127.0.0.1:60248)
0929 16:22:11.759203 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:11.759273 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:38847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260929 16:22:11.759657 16117 catalog_manager.cc:4584] TS 51031fc1b7ef4c4ba8422bdae59728df (127.4.200.65:38847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd37d084d5fa4b368d1789d096df3085: Network error: Client connection negotiation failed: client connection to 127.4.200.65:38847: connect: Connection refused (error 111)
I20260929 16:22:11.764021 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 16485
I20260929 16:22:11.771942 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 16098
I20260929 16:22:11.780112 4897 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:22:12.124966 4897 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:22:12.125097 4897 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:22:12.470254 4897 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:22:12.470 UTC [15684] LOG: received smart shutdown request
2026-09-29 16:22:12.474 UTC [15684] LOG: background worker "logical replication launcher" (PID 15693) exited with exit code 1
2026-09-29 16:22:12.474 UTC [15688] LOG: shutting down
2026-09-29 16:22:12.644 UTC [15684] LOG: database system is shut down
2026-09-29T16:22:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46484 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:22:12 dist-test-slave-s9ct krb5kdc[16670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:22:12 dist-test-slave-s9ct krb5kdc[16670](info): set up 2 sockets
Sep 29 16:22:12 dist-test-slave-s9ct krb5kdc[16670](info): commencing operation
krb5kdc: starting...
W20260929 16:22:13.802620 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:22:13 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:22:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangeradmin_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangeradmin_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangerlookup_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangerlookup_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
I20260929 16:22:13.973004 4897 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh_zHSI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/postgres -l logfile start
2026-09-29 16:22:16.537 UTC [16700] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:22:16.537 UTC [16700] LOG: listening on IPv4 address "127.4.200.84", port 48695
2026-09-29 16:22:16.541 UTC [16700] LOG: listening on Unix socket "/tmp/.s.PGSQL.48695"
2026-09-29 16:22:16.563 UTC [16703] LOG: database system was shut down at 2026-09-29 16:22:15 UTC
2026-09-29 16:22:16.566 UTC [16700] LOG: database system is ready to accept connections
I20260929 16:22:17.534801 4897 mini_postgres.cc:96] Postgres bound to 48695
2026-09-29 16:22:17.541 UTC [16711] FATAL: database "slave" does not exist
127.4.200.84:48695 - accepting connections
I20260929 16:22:17.541935 4897 mini_ranger.cc:161] Starting Ranger...
I20260929 16:22:17.557914 4897 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:22:18.042587 4897 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:22:18.042663 4897 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin
2026-09-29 16:22:18,317 [I] DB FLAVOR :POSTGRES
2026-09-29 16:22:18,317 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:22:18,317 [I] Checking connection..
2026-09-29 16:22:18,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:22:18,569 [I] Checking connection passed.
2026-09-29 16:22:18,569 [I] --------- Verifying version history table ---------
2026-09-29 16:22:18,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:22:18,802 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:22:18,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:22:19,050 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:22:19,050 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:22:19,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:22:19.304 UTC [16775] WARNING: there is no transaction in progress
2026-09-29 16:22:19,310 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:22:19,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:22:19,591 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:22:19,591 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:22:19,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:22:19,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:22:20,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s9ct.c.gcp-upstream.internal','N') ;"
2026-09-29 16:22:20,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:22:20,587 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:22:20,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:22:20,848 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:22:20,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:22:21,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:22:21,357 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:22:21,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:22:22.343 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.361 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.379 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.395 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.411 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.429 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.483 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.489 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.504 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.515 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.525 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.539 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.549 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.556 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.564 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:22.571 UTC [16883] WARNING: there is no transaction in progress
2026-09-29 16:22:23,231 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:22:23,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:22:23,479 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:22:23,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:22:23,733 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:22:23,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:22:23,965 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:22:23,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:22:24,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:22:24,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s9ct.c.gcp-upstream.internal';"
2026-09-29 16:22:24,693 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:22:24,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:22:24,954 [I] DB_PATCHES have already been applied
I20260929 16:22:24.959643 4897 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh_zHSI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:22:24.959715 4897 mini_ranger.cc:193] Using host: 127.4.200.84
I20260929 16:22:24.963263 4897 mini_ranger.cc:227] Ranger admin URL: http://127.4.200.84:57895
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:22:26.078492 4897 mini_ranger.cc:160] Time spent starting Ranger: real 8.537s user 0.000s sys 0.010s
Sep 29, 2026 4:22:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:22:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 4:22:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:22:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:22:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:22:47.997305 4897 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.200.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.200.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangerkms_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.200.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/rangerkms_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.200.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.200.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.200.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/HTTP_127.4.200.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:22:48 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698968, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:22:48.083658 4897 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:22:48.093677 4897 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:22:49.050877 4897 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:22:49.050971 4897 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms
I20260929 16:22:49.050994 4897 mini_ranger_kms.cc:227] Using postgres at 127.4.200.84:48695
2026-09-29 16:22:49,223 [I] DB FLAVOR :POSTGRES
2026-09-29 16:22:49,224 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:22:49,224 [I] Checking connection
2026-09-29 16:22:49,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:22:49,467 [I] connection success
2026-09-29 16:22:49,468 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:22:49,468 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:22:49,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:22:49,732 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:22:49,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:22:49,732 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:22:49,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.200.84:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:22:50,009 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:22:50.505306 4897 mini_ranger_kms.cc:310] Created kms service
I20260929 16:22:50.708061 4897 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:22:50.753113 4897 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:22:50 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698968, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:51.174423 4897 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:22:51.174566 4897 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh_zHSI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh_zHSI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:22:51.174585 4897 mini_ranger_kms.cc:240] Using host: 127.4.200.84
I20260929 16:22:51.178372 4897 mini_ranger_kms.cc:276] Ranger KMS PID: 17063
I20260929 16:22:51.178484 4897 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.200.84:37993
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:22:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:22:53.469120 4897 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.385s user 0.011s sys 0.147s
I20260929 16:22:53.469237 4897 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:22:53.469302 4897 mini_ranger_kms.cc:401] 127.4.200.84:37993/kms/v1/keys
Sep 29 16:22:53 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790698973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:22:54 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
Sep 29 16:22:54 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.200.84@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
Sep 29 16:22:56 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:22:56.680187 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41225
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:41225
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:56.792749 17114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:56.793067 17114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:56.793140 17114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:56.797394 17114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:22:56.797494 17114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:56.797530 17114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:56.797583 17114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:22:56.797619 17114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:22:56.803398 17114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.200.84:37993/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:41225
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:56.804548 17114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:56.805642 17114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:56.814527 17120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:56.814549 17119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:56.814538 17114 server_base.cc:1034] running on GCE node
W20260929 16:22:56.814546 17122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:56.815446 17114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:56.816722 17114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:56.817931 17114 hybrid_clock.cc:648] HybridClock initialized: now 1790698976817902 us; error 50 us; skew 500 ppm
Sep 29 16:22:56 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698976, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:56.826619 17114 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:22:56.827621 17114 webserver.cc:460] Webserver started at http://127.4.200.126:40435/ using document root <none> and password file <none>
I20260929 16:22:56.828239 17114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:56.828325 17114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:56.828552 17114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:56 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698976, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:56.875761 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "05041707a5b94ccfb3a53a2dfc6f7d99"
format_stamp: "Formatted at 2026-09-29 16:22:56 on dist-test-slave-s9ct"
server_key: "51b85b18592c4e35a72c9f284ed9a575"
server_key_iv: "1491d320b17c4cbc04fa96b4f7d971be"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:56.876456 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "05041707a5b94ccfb3a53a2dfc6f7d99"
format_stamp: "Formatted at 2026-09-29 16:22:56 on dist-test-slave-s9ct"
server_key: "51b85b18592c4e35a72c9f284ed9a575"
server_key_iv: "1491d320b17c4cbc04fa96b4f7d971be"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:56.880784 17114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:22:56.895289 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:56.896768 17114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:22:56.896924 17114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "05041707a5b94ccfb3a53a2dfc6f7d99"
format_stamp: "Formatted at 2026-09-29 16:22:56 on dist-test-slave-s9ct"
server_key: "51b85b18592c4e35a72c9f284ed9a575"
server_key_iv: "1491d320b17c4cbc04fa96b4f7d971be"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:56.897097 17114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:56.913130 17114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:56.916038 17114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:56.916249 17114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:56.923727 17114 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:41225
I20260929 16:22:56.923763 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:41225 every 8 connection(s)
I20260929 16:22:56.924707 17114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:22:56.927518 17182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:22:56.928507 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17114
I20260929 16:22:56.928646 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:22:56.928957 4897 external_mini_cluster.cc:1428] Setting key 7b9271327306641f8d06b50264f38f5f
I20260929 16:22:56.933961 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Bootstrap starting.
I20260929 16:22:56.936710 17182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Neither blocks nor log segments found. Creating new log.
I20260929 16:22:56.937708 17182 log.cc:824] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:22:56 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:56.939821 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: No bootstrap required, opened a new log
I20260929 16:22:56.942798 17182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:56.943048 17182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:22:56.943130 17182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05041707a5b94ccfb3a53a2dfc6f7d99, State: Initialized, Role: FOLLOWER
I20260929 16:22:56.943713 17182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:56.943874 17182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:22:56.943962 17182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:22:56.944067 17182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:22:56.944612 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:56.930362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.4.200.126:41225)
0929 16:22:56.930827 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:56.930842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:56.930877 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:22:56.931510 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:56.931527 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:56.931872 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:56.932156 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:56.933071 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:56.934233 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:56.935455 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:56.935949 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:56.936079 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:56.939142 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:56.939147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:56.939161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:56.939187 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:56.941427 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:56.941891 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:56.941902 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:56.941907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:56.942006 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:56.942298 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:56.942303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:56.942305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:56.942777 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:56.942995 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:56.943320 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:22:56.943596 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":137,"threads_started":1}
I20260929 16:22:56.945220 17182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:56.945570 17182 leader_election.cc:302] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05041707a5b94ccfb3a53a2dfc6f7d99; no voters:
I20260929 16:22:56.945881 17182 leader_election.cc:288] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:22:56.945987 17187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:22:56.946215 17187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 LEADER]: Becoming Leader. State: Replica: 05041707a5b94ccfb3a53a2dfc6f7d99, State: Running, Role: LEADER
I20260929 16:22:56.946615 17187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:56.947373 17182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:22:56.947832 17189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05041707a5b94ccfb3a53a2dfc6f7d99. Latest consensus state: current_term: 1 leader_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } } }
I20260929 16:22:56.948047 17189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:56.948343 17188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } } }
I20260929 16:22:56.948486 17188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:56.948508 17193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:22:56.951366 17193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:22:56.952383 17182 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:22:56.959820 17193 catalog_manager.cc:1347] Generated new cluster ID: 4ed2865f05654ae3bf3c2c0fafe7e671
I20260929 16:22:56.959897 17193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:22:56.989621 17193 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:22:56.990859 17193 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:22:56.995548 17193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Generated new TSK 0
I20260929 16:22:56.996484 17193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:22:57.641950 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:57.758888 17227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:57.759136 17227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:57.759178 17227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:57.763633 17227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:57.763690 17227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:57.763778 17227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:22:57.769217 17227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:57.770267 17227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:57.771301 17227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:57.780932 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:57.780938 17235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:57.781592 17232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:57.781669 17227 server_base.cc:1034] running on GCE node
I20260929 16:22:57.782131 17227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:57.782956 17227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:57.784185 17227 hybrid_clock.cc:648] HybridClock initialized: now 1790698977784168 us; error 30 us; skew 500 ppm
Sep 29 16:22:57 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:57.787917 17227 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:22:57.789019 17227 webserver.cc:460] Webserver started at http://127.4.200.65:45737/ using document root <none> and password file <none>
I20260929 16:22:57.789683 17227 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:57.789858 17227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:57.790180 17227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:57 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:57.807611 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "92eb42e417a04ca7ad99141c64e0a603"
format_stamp: "Formatted at 2026-09-29 16:22:57 on dist-test-slave-s9ct"
server_key: "9129e6dbdb6ef435ee86e435c12e1f46"
server_key_iv: "70eef457bdf003dde87937116ceb15c4"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:57.808368 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "92eb42e417a04ca7ad99141c64e0a603"
format_stamp: "Formatted at 2026-09-29 16:22:57 on dist-test-slave-s9ct"
server_key: "9129e6dbdb6ef435ee86e435c12e1f46"
server_key_iv: "70eef457bdf003dde87937116ceb15c4"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:57.818068 17227 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20260929 16:22:57.838361 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:57.840047 17227 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 16:22:57.840214 17227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "92eb42e417a04ca7ad99141c64e0a603"
format_stamp: "Formatted at 2026-09-29 16:22:57 on dist-test-slave-s9ct"
server_key: "9129e6dbdb6ef435ee86e435c12e1f46"
server_key_iv: "70eef457bdf003dde87937116ceb15c4"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:57.840371 17227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:57.870172 17227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:57.874119 17227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:57.874413 17227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:57.875159 17227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:57.876317 17227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:57.876413 17227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:57.876498 17227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:57.876549 17227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:57.887569 17227 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:42787
I20260929 16:22:57.888832 17227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:22:57.889385 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17227
I20260929 16:22:57.889518 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:22:57.889889 4897 external_mini_cluster.cc:1428] Setting key bb03ccf1f144de1fc4acce1feb04356c
I20260929 16:22:57.889863 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:42787 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Sep 29 16:22:57 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:57.920877 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:57.893726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.65:60647)
0929 16:22:57.895221 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:57.895293 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:57.896527 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:57.897477 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:57.897487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:57.897856 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:57.898548 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:57.898577 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:57.899789 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:57.899793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:57.901228 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:57.901239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:57.902048 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:57.902052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:57.902250 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:57.902979 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:57.903002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:57.911632 (+ 8630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:57.917040 (+ 5408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:57.917050 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:57.917085 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:57.917549 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:57.917948 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:57.917952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:57.917954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:57.918186 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:57.918724 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:57.918733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:57.919099 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 16:22:57.919298 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 16:22:57.919710 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"thread_start_us":149,"threads_started":1}
I20260929 16:22:57.922480 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:57.894972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:60647 (local address 127.4.200.126:41225)
0929 16:22:57.897004 (+ 2032us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:57.897015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:57.897032 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:22:57.897078 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:57.897083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:57.897143 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:57.897251 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:57.898748 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:57.899640 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:57.901432 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:57.901866 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:57.903712 (+ 1846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:57.914656 (+ 10944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:57.914661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:57.914665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:57.914705 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:57.916842 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:57.917711 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:57.917714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:57.917718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:57.917821 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:57.918306 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:57.918309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:57.918311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:57.918596 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:57.921982 (+ 3386us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:57.922097 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 16:22:57.922261 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1932,"thread_start_us":129,"threads_started":1}
I20260929 16:22:57.930455 17360 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:57.930763 17360 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:57.931437 17360 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:57.933391 17146 ts_manager.cc:194] Registered new tserver with Master: 92eb42e417a04ca7ad99141c64e0a603 (127.4.200.65:42787)
I20260929 16:22:57.935030 17146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:60647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:22:57.954929 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:22:57 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790698977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:22:58.113724 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:58.114156 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:58.114240 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:58.120671 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:58.120844 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:58.121006 17366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:22:58.129370 17366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:58.130878 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:58.132332 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:58.145255 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:58.145743 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:58.146385 17366 server_base.cc:1034] running on GCE node
W20260929 16:22:58.146389 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:58.147140 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:58.147933 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:58.149171 17366 hybrid_clock.cc:648] HybridClock initialized: now 1790698978149077 us; error 108 us; skew 500 ppm
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:58.153016 17366 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:22:58.154139 17366 webserver.cc:460] Webserver started at http://127.4.200.66:33257/ using document root <none> and password file <none>
I20260929 16:22:58.154870 17366 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:58.154944 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:58.155203 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:58.170451 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "c45d3c17afad42f39c78ba90f4bd2134"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "080a633df8c38e328f4c8a9c5131b924"
server_key_iv: "c2d9d8867e7fdcbcd6e58d3c99df5c3d"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.171303 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "c45d3c17afad42f39c78ba90f4bd2134"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "080a633df8c38e328f4c8a9c5131b924"
server_key_iv: "c2d9d8867e7fdcbcd6e58d3c99df5c3d"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.180835 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260929 16:22:58.213369 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.215372 17366 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.000s
I20260929 16:22:58.215607 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "c45d3c17afad42f39c78ba90f4bd2134"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "080a633df8c38e328f4c8a9c5131b924"
server_key_iv: "c2d9d8867e7fdcbcd6e58d3c99df5c3d"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.215868 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:58.248694 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:58.252094 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:58.252300 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:58.252882 17366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:58.253739 17366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:58.253809 17366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.253909 17366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:58.253962 17366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.263437 17366 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:35159
I20260929 16:22:58.264423 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:22:58.264933 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:35159 every 8 connection(s)
I20260929 16:22:58.275956 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17366
I20260929 16:22:58.276111 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:22:58.276444 4897 external_mini_cluster.cc:1428] Setting key 22204917d2e9a418a566a0b67b1b930e
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:58.301370 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:58.266613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:56699 (local address 127.4.200.126:41225)
0929 16:22:58.266778 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:58.266783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:58.282510 (+ 15727us) server_negotiation.cc:407] Connection header received
0929 16:22:58.284534 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:58.284538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:58.284618 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:58.284679 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:58.286636 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.287451 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.289191 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.289556 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.289638 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:58.296679 (+ 7041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:58.296684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:58.296687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:58.296719 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:58.298587 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:58.299294 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:58.299298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:58.299300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:58.299390 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:58.299870 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:58.299874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:58.299876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:58.300120 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:58.300208 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:58.301018 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 16:22:58.301160 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:22:58.303412 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:58.267069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.66:56699)
0929 16:22:58.282275 (+ 15206us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:58.282370 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:58.284189 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:58.285264 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:58.285277 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:58.285680 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:58.286444 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:58.286469 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.287623 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.287626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:58.289044 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:58.289055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.289862 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.289867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:58.290069 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:58.294563 (+ 4494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:58.294590 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:58.296428 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:58.298742 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:58.298749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:58.298769 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:58.299149 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:58.299503 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:58.299506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:58.299508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:58.299746 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:58.300292 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:58.300298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:58.300659 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 16:22:58.301875 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0929 16:22:58.302264 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14874,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:22:58.305083 17496 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:58.305346 17496 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:58.305944 17496 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:58.307214 17146 ts_manager.cc:194] Registered new tserver with Master: c45d3c17afad42f39c78ba90f4bd2134 (127.4.200.66:35159)
I20260929 16:22:58.308005 17146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:56699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:22:58.334771 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
W20260929 16:22:58.524283 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:58.524820 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:58.524952 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:58.531019 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:58.531229 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:58.531379 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:22:58.539270 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:58.540957 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:58.542485 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:58.556233 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:58.556267 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:58.556856 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:58.557601 17501 server_base.cc:1034] running on GCE node
I20260929 16:22:58.558090 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:58.558964 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:58.590592 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790698978590467 us; error 153 us; skew 500 ppm
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:58.594853 17501 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:22:58.596160 17501 webserver.cc:460] Webserver started at http://127.4.200.67:36253/ using document root <none> and password file <none>
I20260929 16:22:58.596964 17501 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:58.597106 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:22:58.597406 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:58.623930 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "96dd40c4815b4dc3b2137625501d6328"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "16cb7e2e6ae9aa7b782cb989afde0a71"
server_key_iv: "5ff22611183de208bcbdb511b172daa6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.624691 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "96dd40c4815b4dc3b2137625501d6328"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "16cb7e2e6ae9aa7b782cb989afde0a71"
server_key_iv: "5ff22611183de208bcbdb511b172daa6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.629992 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:22:58.660953 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.662863 17501 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 16:22:58.663033 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "96dd40c4815b4dc3b2137625501d6328"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "16cb7e2e6ae9aa7b782cb989afde0a71"
server_key_iv: "5ff22611183de208bcbdb511b172daa6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.663192 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:58.689898 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:58.701289 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:58.701555 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:58.702234 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:58.703753 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:58.703815 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.703884 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:58.703917 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.715853 17501 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:34345
I20260929 16:22:58.717023 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:22:58.721124 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17501
I20260929 16:22:58.721253 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:22:58.721546 4897 external_mini_cluster.cc:1428] Setting key 3ce1540440c38051520693a385f4205b
I20260929 16:22:58.724143 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:34345 every 8 connection(s)
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:58.752099 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:58.734321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:52251 (local address 127.4.200.126:41225)
0929 16:22:58.734488 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:58.734492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:58.738747 (+ 4255us) server_negotiation.cc:407] Connection header received
0929 16:22:58.740022 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:58.740026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:58.740082 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:58.740130 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:58.741809 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.742624 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.744242 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.744561 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.744632 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:58.748139 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:58.748143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:58.748146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:58.748172 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:58.749781 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:58.750353 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:58.750356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:58.750358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:58.750473 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:58.750853 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:58.750856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:58.750858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:58.751090 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:58.751172 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:58.751829 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 16:22:58.751943 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:22:58.753911 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:58.733065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.67:52251)
0929 16:22:58.738561 (+ 5496us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:58.738634 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:58.739758 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:58.740571 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:58.740581 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:58.740965 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:58.741636 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:58.741654 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.742785 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.742789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:58.744112 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:58.744122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:58.744860 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:58.744863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:58.745046 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:58.745752 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:58.745772 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:58.747954 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:58.749905 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:58.749911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:58.749927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:58.750245 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:58.750577 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:58.750580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:58.750582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:58.750755 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:58.751253 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:58.751258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:58.751552 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 16:22:58.752495 (+ 943us) client_negotiation.cc:240] Negotiation successful
0929 16:22:58.752842 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5191,"thread_start_us":200,"threads_started":1}
I20260929 16:22:58.755452 17630 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:58.755739 17630 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:58.756327 17630 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:58.757522 17146 ts_manager.cc:194] Registered new tserver with Master: 96dd40c4815b4dc3b2137625501d6328 (127.4.200.67:34345)
I20260929 16:22:58.758266 17146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:52251
I20260929 16:22:58.761318 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:22:58.762091 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17227
I20260929 16:22:58.768440 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17366
I20260929 16:22:58.775833 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17501
I20260929 16:22:58.782653 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17114
I20260929 16:22:58.789948 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41225
--webserver_interface=127.4.200.126
--webserver_port=40435
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:41225
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:22:58.903673 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:58.904021 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:58.904091 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:58.908314 17633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:22:58.908399 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:58.908438 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:58.908491 17633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:22:58.908529 17633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:22:58.914165 17633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:41225
--ranger_config_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:58.915269 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:58.916312 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:58.924877 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:58.924890 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:58.925302 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:58.925151 17633 server_base.cc:1034] running on GCE node
I20260929 16:22:58.925968 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:58.927142 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:58.928336 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790698978928297 us; error 61 us; skew 500 ppm
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:58.931097 17633 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:22:58.932133 17633 webserver.cc:460] Webserver started at http://127.4.200.126:40435/ using document root <none> and password file <none>
I20260929 16:22:58.932709 17633 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:58.932781 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:22:58 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:58.990618 17633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:22:58.992937 17649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:58.994305 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:22:58.994486 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "05041707a5b94ccfb3a53a2dfc6f7d99"
format_stamp: "Formatted at 2026-09-29 16:22:56 on dist-test-slave-s9ct"
server_key: "51b85b18592c4e35a72c9f284ed9a575"
server_key_iv: "1491d320b17c4cbc04fa96b4f7d971be"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:58.995066 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:59.014024 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:59.017059 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:59.017287 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:59.025540 17633 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:41225
I20260929 16:22:59.025540 17701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:41225 every 8 connection(s)
I20260929 16:22:59.026852 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:22:59.028124 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17633
I20260929 16:22:59.028601 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:42787
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:41225
--webserver_port=45737
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:22:59.031921 17702 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:22:59.034497 17702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Bootstrap starting.
I20260929 16:22:59.057196 17702 log.cc:824] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Log is configured to *not* fsync() on all Append() calls
I20260929 16:22:59.065840 17702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:22:59.066327 17702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Bootstrap complete.
I20260929 16:22:59.071789 17702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:59.072304 17702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05041707a5b94ccfb3a53a2dfc6f7d99, State: Initialized, Role: FOLLOWER
I20260929 16:22:59.072937 17702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:59.073122 17702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:22:59.073227 17702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:22:59.073338 17702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:22:59.075853 17702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:59.076213 17702 leader_election.cc:302] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05041707a5b94ccfb3a53a2dfc6f7d99; no voters:
I20260929 16:22:59.076563 17702 leader_election.cc:288] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:22:59.076674 17706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:22:59.076972 17706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [term 2 LEADER]: Becoming Leader. State: Replica: 05041707a5b94ccfb3a53a2dfc6f7d99, State: Running, Role: LEADER
I20260929 16:22:59.077332 17706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } }
I20260929 16:22:59.077901 17702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:22:59.078364 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } } }
I20260929 16:22:59.078581 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:59.078814 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05041707a5b94ccfb3a53a2dfc6f7d99. Latest consensus state: current_term: 2 leader_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05041707a5b94ccfb3a53a2dfc6f7d99" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 41225 } } }
I20260929 16:22:59.078933 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:22:59.079213 17710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:22:59.082895 17710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:22:59.083454 17702 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:22:59.083781 17710 catalog_manager.cc:1259] Loaded cluster ID: 4ed2865f05654ae3bf3c2c0fafe7e671
I20260929 16:22:59.083870 17710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:22:59.086277 17710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:22:59.086974 17710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Loaded TSK: 0
I20260929 16:22:59.088209 17710 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:22:59.188958 17704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:59.189325 17704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:59.189401 17704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:59.195932 17704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:59.196015 17704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:59.196108 17704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:22:59.204844 17704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:42787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=45737
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:59.206291 17704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:59.207717 17704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:59.221840 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:59.222549 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:59.222764 17704 server_base.cc:1034] running on GCE node
W20260929 16:22:59.223057 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:59.223531 17704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:59.224313 17704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:59.225536 17704 hybrid_clock.cc:648] HybridClock initialized: now 1790698979225501 us; error 48 us; skew 500 ppm
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:59.229175 17704 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:22:59.230264 17704 webserver.cc:460] Webserver started at http://127.4.200.65:45737/ using document root <none> and password file <none>
I20260929 16:22:59.230993 17704 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:59.231097 17704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:59.264068 17704 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 16:22:59.267827 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.269605 17704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 16:22:59.269744 17704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "92eb42e417a04ca7ad99141c64e0a603"
format_stamp: "Formatted at 2026-09-29 16:22:57 on dist-test-slave-s9ct"
server_key: "9129e6dbdb6ef435ee86e435c12e1f46"
server_key_iv: "70eef457bdf003dde87937116ceb15c4"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:59.270565 17704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:59.291678 17704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:59.296406 17704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:59.296756 17704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:59.297616 17704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:59.299027 17704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:59.299153 17704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.299306 17704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:59.299408 17704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.314575 17704 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:42787
I20260929 16:22:59.314745 17863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:42787 every 8 connection(s)
I20260929 16:22:59.315810 17704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:22:59.316598 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17704
I20260929 16:22:59.317093 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:35159
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:41225
--webserver_port=33257
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:59.364415 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.65:56949)
0929 16:22:59.326776 (+ 7840us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:59.326849 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:59.328230 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:59.330635 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:59.330643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:59.330975 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:59.331566 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.331593 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.344615 (+ 13022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.344620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.345946 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.345963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.355560 (+ 9597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.355565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.355794 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.356523 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:59.356544 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:59.358304 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:59.360872 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.360878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.360892 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.361285 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.361737 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.361740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.361742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.361902 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.362669 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:59.362675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:59.362953 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 16:22:59.363135 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:22:59.363451 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7533,"thread_start_us":163,"threads_started":1}
I20260929 16:22:59.366493 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.320522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56949 (local address 127.4.200.126:41225)
0929 16:22:59.328841 (+ 8319us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:59.328866 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:59.328916 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:22:59.329245 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:59.329263 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:59.329748 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:59.330112 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:59.342583 (+ 12471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.344386 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.354511 (+ 10125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.355291 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.356251 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.358601 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:59.358608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:59.358625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:59.358657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:59.360713 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.361455 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.361462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.361468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.361620 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.362014 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.362019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.362023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.362522 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:59.365149 (+ 2627us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:59.365307 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:22:59.365616 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8092,"thread_start_us":127,"threads_started":1}
I20260929 16:22:59.369199 17864 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:59.369457 17864 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.370034 17864 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.372275 17864 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:22:59.372406 17864 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.372601 17864 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:59.373487 17864 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.373684 17864 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.374365 17864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:22:59.471565 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:59.471865 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:59.471936 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:59.477496 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:59.477576 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:59.477664 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:22:59.485595 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:35159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=33257
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:59.486999 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:59.488410 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:59.500409 17873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:59.500409 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:59.500545 17874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:59.500900 17866 server_base.cc:1034] running on GCE node
I20260929 16:22:59.501358 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:59.502079 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:59.503254 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790698979503236 us; error 34 us; skew 500 ppm
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:59.506701 17866 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:22:59.507849 17866 webserver.cc:460] Webserver started at http://127.4.200.66:33257/ using document root <none> and password file <none>
I20260929 16:22:59.508562 17866 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:59.508666 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:59.528733 17866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:22:59.531940 17884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.533588 17866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:22:59.533716 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "c45d3c17afad42f39c78ba90f4bd2134"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "080a633df8c38e328f4c8a9c5131b924"
server_key_iv: "c2d9d8867e7fdcbcd6e58d3c99df5c3d"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:59.534461 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:59.555002 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:59.558758 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:59.559001 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:59.559741 17866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:59.560773 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:59.560830 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.560889 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:59.560914 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.573177 17866 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:35159
I20260929 16:22:59.573251 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:35159 every 8 connection(s)
I20260929 16:22:59.574713 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:22:59.582695 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 17866
I20260929 16:22:59.583225 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.200.84:37993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:34345
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:41225
--webserver_port=36253
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:59.601178 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.581383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:48075 (local address 127.4.200.126:41225)
0929 16:22:59.581544 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:59.581549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:59.586984 (+ 5435us) server_negotiation.cc:407] Connection header received
0929 16:22:59.588217 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:59.588221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:59.588280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:59.588381 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:59.590435 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.591308 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.593045 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.593422 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.593501 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.596749 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:59.596753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:59.596756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:59.596785 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:59.598583 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.599270 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.599273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.599275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.599359 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.599767 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.599774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.599776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.600026 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:59.600117 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:59.600853 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 16:22:59.600992 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:22:59.602895 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.580129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.66:48075)
0929 16:22:59.586764 (+ 6635us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:59.586832 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:59.587971 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:59.589128 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:59.589139 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:59.589524 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:59.590218 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.590238 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.591473 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.591476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.592883 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.592897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.593716 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.593720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.593927 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.594679 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:59.594699 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:59.596543 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:59.598733 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.598739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.598757 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.599156 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.599462 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.599465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.599467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.599638 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.600235 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:59.600242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:59.600561 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 16:22:59.601661 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0929 16:22:59.601992 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6367,"thread_start_us":142,"threads_started":1}
I20260929 16:22:59.604038 17998 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:59.604265 17998 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.604755 17998 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.606320 17998 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:22:59.606470 17998 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.606627 17998 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:59.607398 17998 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.607542 17998 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.608304 17998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:22:59.738039 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:22:59.738420 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:22:59.738489 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:22:59.744804 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:22:59.744894 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:22:59.744982 18001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:22:59.751847 18001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.200.84:37993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:34345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=36253
--tserver_master_addrs=127.4.200.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:22:59.752959 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:22:59.754004 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:22:59.764747 18010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:59.764982 18001 server_base.cc:1034] running on GCE node
W20260929 16:22:59.764744 18008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:22:59.764761 18007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:22:59.765590 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:22:59.766216 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:22:59.767401 18001 hybrid_clock.cc:648] HybridClock initialized: now 1790698979767280 us; error 135 us; skew 500 ppm
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:22:59.770457 18001 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:22:59.771476 18001 webserver.cc:460] Webserver started at http://127.4.200.67:36253/ using document root <none> and password file <none>
I20260929 16:22:59.772221 18001 fs_manager.cc:359] Metadata directory not provided
I20260929 16:22:59.772342 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:22:59.790218 18001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:22:59.797094 18018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.798943 18001 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 16:22:59.799098 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "96dd40c4815b4dc3b2137625501d6328"
format_stamp: "Formatted at 2026-09-29 16:22:58 on dist-test-slave-s9ct"
server_key: "16cb7e2e6ae9aa7b782cb989afde0a71"
server_key_iv: "5ff22611183de208bcbdb511b172daa6"
server_key_version: "kuduclusterkey@0"
I20260929 16:22:59.799875 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:22:59.818634 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:22:59.822618 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:22:59.822826 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:22:59.823550 18001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:22:59.824615 18001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:22:59.824692 18001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.824764 18001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:22:59.824817 18001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:22:59.836778 18001 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:34345
I20260929 16:22:59.838245 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:22:59.838635 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:34345 every 8 connection(s)
I20260929 16:22:59.839612 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18001
I20260929 16:22:59.850256 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.840430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.4.200.126:41225)
0929 16:22:59.840587 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:59.840593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:59.840969 (+ 376us) server_negotiation.cc:407] Connection header received
0929 16:22:59.841486 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:59.841489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:59.841545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:59.841660 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:59.842601 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.843419 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.844597 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.844929 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.845039 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.846635 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:59.846638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:59.846641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:59.846667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:59.848413 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.848842 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.848850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.848851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.848924 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.849197 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.849202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.849203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.849380 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:59.849620 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:59.849821 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:22:59.849927 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 16:22:59 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:22:59.857183 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.842716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:57899 (local address 127.4.200.126:41225)
0929 16:22:59.842945 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:22:59.842956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:22:59.843195 (+ 239us) server_negotiation.cc:407] Connection header received
0929 16:22:59.844350 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:22:59.844353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:22:59.844403 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:22:59.844500 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:22:59.845831 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.847036 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.849925 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.850264 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.850366 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.853198 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:22:59.853202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:22:59.853204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:22:59.853248 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:22:59.854797 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.855405 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.855408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.855410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.855505 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:22:59.855917 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:22:59.855921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:22:59.855923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:22:59.856141 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:22:59.856241 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:22:59.856874 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:22:59.857002 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260929 16:22:59.858220 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:22:59.841597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:41225 (local address 127.4.200.67:57899)
0929 16:22:59.843070 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0929 16:22:59.843110 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:22:59.844110 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:22:59.844707 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:22:59.844714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:22:59.845005 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:22:59.845658 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.845691 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.847183 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.847187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.849780 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:22:59.849791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:22:59.850553 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:22:59.850557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:22:59.850768 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:22:59.851322 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:22:59.851343 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:22:59.853018 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:22:59.854916 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.854922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.854938 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.855287 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.855622 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:22:59.855625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:22:59.855627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:22:59.855806 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:22:59.856264 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:22:59.856271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:22:59.856592 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 16:22:59.856814 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 16:22:59.857144 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":133,"threads_started":1}
I20260929 16:22:59.859490 18132 heartbeater.cc:347] Connected to a master server at 127.4.200.126:41225
I20260929 16:22:59.859745 18132 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.860282 18132 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.862030 18132 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:22:59.862152 18132 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.862314 18132 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:22:59.863818 18132 heartbeater.cc:464] Registering TS with master...
I20260929 16:22:59.863948 18132 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
W20260929 16:22:59.864688 18132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:23:00 dist-test-slave-s9ct krb5kdc[16670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790698980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:00.374622 17864 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:00.374917 17864 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:23:00.376199 17666 ts_manager.cc:194] Registered new tserver with Master: 92eb42e417a04ca7ad99141c64e0a603 (127.4.200.65:42787)
I20260929 16:23:00.378017 17666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:56949
I20260929 16:23:00.608500 17998 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:00.608805 17998 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:23:00.609561 17665 ts_manager.cc:194] Registered new tserver with Master: c45d3c17afad42f39c78ba90f4bd2134 (127.4.200.66:35159)
I20260929 16:23:00.610351 17665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:48075
Sep 29 16:23:00 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:23:00.864933 18132 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:00.865233 18132 heartbeater.cc:511] Master 127.4.200.126:41225 requested a full tablet report, sending...
I20260929 16:23:00.865980 17665 ts_manager.cc:194] Registered new tserver with Master: 96dd40c4815b4dc3b2137625501d6328 (127.4.200.67:34345)
I20260929 16:23:00.866855 17665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:57899
I20260929 16:23:00.869654 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:23:00.882179 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.871336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.4.200.126:41225)
0929 16:23:00.871919 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:00.871924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:00.872281 (+ 357us) server_negotiation.cc:407] Connection header received
0929 16:23:00.872324 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:00.872327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:00.872375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:00.872460 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:00.873406 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.874197 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.875410 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.875826 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.875925 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.877869 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:00.877873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:00.877876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:00.877903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:00.879887 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:00.880433 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:00.880437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:00.880438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:00.880528 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:00.880842 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:00.880845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:00.880846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:00.881053 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:00.881178 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:00.881871 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 16:23:00.881997 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":100,"threads_started":1}
I20260929 16:23:00.886844 17665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:23:00.891326 17665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 16:23:00 dist-test-slave-s9ct krb5kdc[16670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for HTTP/127.4.200.84@KRBTEST.COM
I20260929 16:23:00.930580 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.913957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:34345 (local address 127.0.0.1:58236)
0929 16:23:00.914810 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:00.914858 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:00.914979 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:00.923153 (+ 8174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:00.923160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:00.923180 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:00.923368 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.923375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.925036 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.925039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.926426 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.926437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.929991 (+ 3554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.929994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.930092 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.930138 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:23:00.930234 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:23:00.930424 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260929 16:23:00.930711 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.915421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.4.200.65:42787)
0929 16:23:00.915951 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:00.915959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:00.923023 (+ 7064us) server_negotiation.cc:407] Connection header received
0929 16:23:00.923125 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:00.923131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:00.923340 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:00.923498 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:00.924168 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.926250 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.927854 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.928662 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.929623 (+ 961us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.929791 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:00.930417 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 16:23:00.930543 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":117,"threads_started":1}
I20260929 16:23:00.931509 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.914179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35159 (local address 127.0.0.1:50870)
0929 16:23:00.921830 (+ 7651us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:00.921844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:00.921953 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:00.923473 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:00.923476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:00.923492 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:00.923564 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.923569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.925036 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.925039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.927603 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.927615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.929808 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.929812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.931224 (+ 1412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.931248 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:23:00.931314 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:23:00.931380 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7581,"mutex_wait_us":1270,"thread_start_us":91,"threads_started":1}
I20260929 16:23:00.932297 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.914699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.4.200.67:34345)
0929 16:23:00.922208 (+ 7509us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:00.922215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:00.922257 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:23:00.922346 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:00.922353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:00.922641 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:00.923519 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:00.923673 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.924902 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.927756 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.928572 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.931853 (+ 3281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.932011 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:00.932060 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 16:23:00.932157 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7392,"thread_start_us":93,"threads_started":1}
I20260929 16:23:00.932636 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.915240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:42787 (local address 127.0.0.1:36502)
0929 16:23:00.922895 (+ 7655us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:00.922909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:00.923006 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:00.923677 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:00.923681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:00.923699 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:00.923796 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.923806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.926433 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.926438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.927707 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:00.927717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.929895 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.929899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:00.930058 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.930131 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 16:23:00.930259 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:23:00.932525 (+ 2266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7588,"mutex_wait_us":2183,"thread_start_us":98,"threads_started":1}
I20260929 16:23:00.933061 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:00.918541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.4.200.66:35159)
0929 16:23:00.918904 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:00.918911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:00.922869 (+ 3958us) server_negotiation.cc:407] Connection header received
0929 16:23:00.922963 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:00.922969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:00.923175 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:00.923324 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:00.923685 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.924902 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.928695 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:00.929481 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:00.929582 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:00.929735 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:00.932820 (+ 3085us) server_negotiation.cc:299] Negotiation successful
0929 16:23:00.932928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260929 16:23:00.937129 17914 tablet_service.cc:1467] Processing CreateTablet for tablet 31682432f687479bab4d2494ea11885a (DEFAULT_TABLE table=test-table [id=52c43364bdb54f549ac27b9081cd9ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:00.937371 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 31682432f687479bab4d2494ea11885a (DEFAULT_TABLE table=test-table [id=52c43364bdb54f549ac27b9081cd9ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:00.938328 17914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31682432f687479bab4d2494ea11885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:00.938376 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31682432f687479bab4d2494ea11885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:00.942687 17794 tablet_service.cc:1467] Processing CreateTablet for tablet 31682432f687479bab4d2494ea11885a (DEFAULT_TABLE table=test-table [id=52c43364bdb54f549ac27b9081cd9ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:00.943713 17794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31682432f687479bab4d2494ea11885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:00.949479 18167 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Bootstrap starting.
I20260929 16:23:00.951864 18167 tablet_bootstrap.cc:654] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:00.953452 18166 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Bootstrap starting.
I20260929 16:23:00.952775 18167 log.cc:824] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:00.956071 18166 tablet_bootstrap.cc:654] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:00.956431 18167 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: No bootstrap required, opened a new log
I20260929 16:23:00.956655 18167 ts_tablet_manager.cc:1397] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:23:00.956923 18166 log.cc:824] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:00.957657 18168 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Bootstrap starting.
I20260929 16:23:00.959787 18168 tablet_bootstrap.cc:654] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:00.960561 18168 log.cc:824] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:00.960978 18167 raft_consensus.cc:348] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.961272 18167 raft_consensus.cc:374] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:00.961336 18167 raft_consensus.cc:729] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96dd40c4815b4dc3b2137625501d6328, State: Initialized, Role: FOLLOWER
I20260929 16:23:00.962855 18167 consensus_queue.cc:260] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.962960 18166 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: No bootstrap required, opened a new log
I20260929 16:23:00.963132 18166 ts_tablet_manager.cc:1397] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 16:23:00.963872 18167 ts_tablet_manager.cc:1428] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:23:00.964232 18132 heartbeater.cc:503] Master 127.4.200.126:41225 was elected leader, sending a full tablet report...
I20260929 16:23:00.967170 18166 raft_consensus.cc:348] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.967463 18166 raft_consensus.cc:374] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:00.967548 18166 raft_consensus.cc:729] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45d3c17afad42f39c78ba90f4bd2134, State: Initialized, Role: FOLLOWER
I20260929 16:23:00.968021 18166 consensus_queue.cc:260] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.969008 18166 ts_tablet_manager.cc:1428] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:00.969380 17998 heartbeater.cc:503] Master 127.4.200.126:41225 was elected leader, sending a full tablet report...
I20260929 16:23:00.971678 18168 tablet_bootstrap.cc:492] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: No bootstrap required, opened a new log
I20260929 16:23:00.971895 18168 ts_tablet_manager.cc:1397] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260929 16:23:00.976076 18168 raft_consensus.cc:348] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.976346 18168 raft_consensus.cc:374] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:00.976409 18168 raft_consensus.cc:729] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92eb42e417a04ca7ad99141c64e0a603, State: Initialized, Role: FOLLOWER
I20260929 16:23:00.976908 18168 consensus_queue.cc:260] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:00.977955 18168 ts_tablet_manager.cc:1428] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:23:00.978374 17864 heartbeater.cc:503] Master 127.4.200.126:41225 was elected leader, sending a full tablet report...
W20260929 16:23:01.070438 17865 tablet.cc:2369] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:23:01.079986 17999 tablet.cc:2369] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:01.081367 18175 raft_consensus.cc:482] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:01.081573 18175 raft_consensus.cc:504] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:01.082934 18175 leader_election.cc:288] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96dd40c4815b4dc3b2137625501d6328 (127.4.200.67:34345), c45d3c17afad42f39c78ba90f4bd2134 (127.4.200.66:35159)
W20260929 16:23:01.098793 18133 tablet.cc:2369] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:01.100075 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.084376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:60285 (local address 127.4.200.67:34345)
0929 16:23:01.084681 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:01.084687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:01.085087 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:23:01.085305 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:01.085309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:01.085358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:01.085407 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:01.086216 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.087057 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.097388 (+ 10331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.098047 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.098188 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:01.098249 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:01.099834 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0929 16:23:01.099934 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260929 16:23:01.100493 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.083428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:34345 (local address 127.4.200.65:60285)
0929 16:23:01.084994 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:01.085009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:01.085174 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:01.085571 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:01.085574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:01.085658 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:01.085810 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:01.085817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.093484 (+ 7667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.093488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:01.094963 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:01.094979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.099483 (+ 4504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.099487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:01.099657 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:01.099686 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:23:01.100235 (+ 549us) client_negotiation.cc:240] Negotiation successful
0929 16:23:01.100332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"thread_start_us":133,"threads_started":1}
I20260929 16:23:01.101254 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31682432f687479bab4d2494ea11885a" candidate_uuid: "92eb42e417a04ca7ad99141c64e0a603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dd40c4815b4dc3b2137625501d6328" is_pre_election: true
I20260929 16:23:01.101613 18077 raft_consensus.cc:2454] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92eb42e417a04ca7ad99141c64e0a603 in term 0.
I20260929 16:23:01.101917 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.084503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:43333 (local address 127.4.200.66:35159)
0929 16:23:01.084782 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:01.084787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:01.086134 (+ 1347us) server_negotiation.cc:407] Connection header received
0929 16:23:01.095182 (+ 9048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:01.095186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:01.095251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:01.095305 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:01.095744 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.096478 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.100484 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.101099 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.101194 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:01.101244 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:01.101710 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 16:23:01.101795 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 16:23:01.102176 17751 leader_election.cc:302] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92eb42e417a04ca7ad99141c64e0a603, 96dd40c4815b4dc3b2137625501d6328; no voters:
I20260929 16:23:01.102325 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.083900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:35159 (local address 127.4.200.65:43333)
0929 16:23:01.086051 (+ 2151us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:01.086066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:01.093268 (+ 7202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:01.095454 (+ 2186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:01.095456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:01.095479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:01.095593 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:01.095600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.096629 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.096646 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:01.100072 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:01.100084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.101446 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.101449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:01.101547 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:01.101568 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:23:01.102107 (+ 539us) client_negotiation.cc:240] Negotiation successful
0929 16:23:01.102193 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"thread_start_us":106,"threads_started":1}
I20260929 16:23:01.102538 18175 raft_consensus.cc:2781] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:01.102632 18175 raft_consensus.cc:482] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:01.102691 18175 raft_consensus.cc:3037] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:01.102947 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31682432f687479bab4d2494ea11885a" candidate_uuid: "92eb42e417a04ca7ad99141c64e0a603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d3c17afad42f39c78ba90f4bd2134" is_pre_election: true
I20260929 16:23:01.103233 17951 raft_consensus.cc:2454] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92eb42e417a04ca7ad99141c64e0a603 in term 0.
I20260929 16:23:01.104120 18175 raft_consensus.cc:504] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:01.104552 18175 leader_election.cc:288] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [CANDIDATE]: Term 1 election: Requested vote from peers 96dd40c4815b4dc3b2137625501d6328 (127.4.200.67:34345), c45d3c17afad42f39c78ba90f4bd2134 (127.4.200.66:35159)
I20260929 16:23:01.107779 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31682432f687479bab4d2494ea11885a" candidate_uuid: "92eb42e417a04ca7ad99141c64e0a603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d3c17afad42f39c78ba90f4bd2134"
I20260929 16:23:01.107975 17951 raft_consensus.cc:3037] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:01.108755 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31682432f687479bab4d2494ea11885a" candidate_uuid: "92eb42e417a04ca7ad99141c64e0a603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dd40c4815b4dc3b2137625501d6328"
I20260929 16:23:01.108908 18077 raft_consensus.cc:3037] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:01.109515 17951 raft_consensus.cc:2454] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92eb42e417a04ca7ad99141c64e0a603 in term 1.
I20260929 16:23:01.109941 17753 leader_election.cc:302] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92eb42e417a04ca7ad99141c64e0a603, c45d3c17afad42f39c78ba90f4bd2134; no voters:
I20260929 16:23:01.110184 18175 raft_consensus.cc:2781] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:01.110324 18077 raft_consensus.cc:2454] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92eb42e417a04ca7ad99141c64e0a603 in term 1.
I20260929 16:23:01.110472 18175 raft_consensus.cc:686] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [term 1 LEADER]: Becoming Leader. State: Replica: 92eb42e417a04ca7ad99141c64e0a603, State: Running, Role: LEADER
I20260929 16:23:01.110918 18175 consensus_queue.cc:237] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } }
I20260929 16:23:01.115868 17665 catalog_manager.cc:5495] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92eb42e417a04ca7ad99141c64e0a603 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "92eb42e417a04ca7ad99141c64e0a603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92eb42e417a04ca7ad99141c64e0a603" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 42787 } health_report { overall_health: HEALTHY } } }
I20260929 16:23:01.168144 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.163296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.4.200.65:42787)
0929 16:23:01.163410 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:01.163413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:01.163425 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:23:01.163497 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:01.163499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:01.163550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:01.163674 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:01.163930 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.164606 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.165917 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:01.166256 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:01.166362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:01.166816 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:01.167341 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:01.167457 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:01.167924 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 16:23:01.167986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:23:01.175370 17951 raft_consensus.cc:1261] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 1 FOLLOWER]: Refusing update from remote peer 92eb42e417a04ca7ad99141c64e0a603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:01.175519 18077 raft_consensus.cc:1261] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 1 FOLLOWER]: Refusing update from remote peer 92eb42e417a04ca7ad99141c64e0a603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:01.176311 18178 consensus_queue.cc:1035] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96dd40c4815b4dc3b2137625501d6328" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:01.182433 18178 consensus_queue.cc:1035] T 31682432f687479bab4d2494ea11885a P 92eb42e417a04ca7ad99141c64e0a603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45d3c17afad42f39c78ba90f4bd2134" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 35159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:01.187494 18181 mvcc.cc:205] Tried to move back new op lower bound from 7334703026885025792 to 7334703026638704640. Current Snapshot: MvccSnapshot[applied={T|T < 7334703026885025792}]
I20260929 16:23:01.191228 18184 mvcc.cc:205] Tried to move back new op lower bound from 7334703026885025792 to 7334703026638704640. Current Snapshot: MvccSnapshot[applied={T|T < 7334703026885025792}]
I20260929 16:23:01.196183 18186 mvcc.cc:205] Tried to move back new op lower bound from 7334703026885025792 to 7334703026638704640. Current Snapshot: MvccSnapshot[applied={T|T < 7334703026885025792}]
I20260929 16:23:01.199625 17665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:01.199807 17665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:01.202442 17665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05041707a5b94ccfb3a53a2dfc6f7d99: Sending DeleteTablet for 3 replicas of tablet 31682432f687479bab4d2494ea11885a
I20260929 16:23:01.203472 17794 tablet_service.cc:1514] Processing DeleteTablet for tablet 31682432f687479bab4d2494ea11885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:01 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:36502
I20260929 16:23:01.203976 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 31682432f687479bab4d2494ea11885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:01 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:58236
I20260929 16:23:01.204201 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17704
I20260929 16:23:01.204414 17914 tablet_service.cc:1514] Processing DeleteTablet for tablet 31682432f687479bab4d2494ea11885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:01 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:50870
I20260929 16:23:01.206923 18199 tablet_replica.cc:330] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: stopping tablet replica
I20260929 16:23:01.207237 18199 raft_consensus.cc:2229] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:01.207557 18199 raft_consensus.cc:2258] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:01.208331 18198 tablet_replica.cc:330] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: stopping tablet replica
I20260929 16:23:01.208621 18198 raft_consensus.cc:2229] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:01.208894 18199 ts_tablet_manager.cc:1905] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:01.208906 18198 raft_consensus.cc:2258] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:01.210453 18198 ts_tablet_manager.cc:1905] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:01.211783 18199 ts_tablet_manager.cc:1918] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:01.211898 18199 log.cc:1196] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/31682432f687479bab4d2494ea11885a
I20260929 16:23:01.212247 18199 ts_tablet_manager.cc:1939] T 31682432f687479bab4d2494ea11885a P c45d3c17afad42f39c78ba90f4bd2134: Deleting consensus metadata
I20260929 16:23:01.213321 18198 ts_tablet_manager.cc:1918] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:01.213527 17652 catalog_manager.cc:4841] TS c45d3c17afad42f39c78ba90f4bd2134 (127.4.200.66:35159): tablet 31682432f687479bab4d2494ea11885a (table test-table [id=52c43364bdb54f549ac27b9081cd9ca1]) successfully deleted
I20260929 16:23:01.213627 18198 log.cc:1196] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/31682432f687479bab4d2494ea11885a
I20260929 16:23:01.214004 18198 ts_tablet_manager.cc:1939] T 31682432f687479bab4d2494ea11885a P 96dd40c4815b4dc3b2137625501d6328: Deleting consensus metadata
W20260929 16:23:01.214860 17651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:42787 (error 108)
I20260929 16:23:01.215276 17650 catalog_manager.cc:4841] TS 96dd40c4815b4dc3b2137625501d6328 (127.4.200.67:34345): tablet 31682432f687479bab4d2494ea11885a (table test-table [id=52c43364bdb54f549ac27b9081cd9ca1]) successfully deleted
I20260929 16:23:01.215727 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17866
I20260929 16:23:01.215950 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:01.215639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:42787 (local address 127.0.0.1:36520)
0929 16:23:01.215796 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:01.215865 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:42787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:23:01.218971 17651 catalog_manager.cc:4584] TS 92eb42e417a04ca7ad99141c64e0a603 (127.4.200.65:42787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31682432f687479bab4d2494ea11885a: Network error: Client connection negotiation failed: client connection to 127.4.200.65:42787: connect: Connection refused (error 111)
I20260929 16:23:01.221971 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18001
I20260929 16:23:01.227792 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 17633
I20260929 16:23:01.234467 4897 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:23:01.569588 4897 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:23:01.569725 4897 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:23:01.914700 4897 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:23:01.914 UTC [16700] LOG: received smart shutdown request
2026-09-29 16:23:01.918 UTC [16700] LOG: background worker "logical replication launcher" (PID 16709) exited with exit code 1
2026-09-29 16:23:01.919 UTC [16704] LOG: shutting down
2026-09-29 16:23:02.103 UTC [16700] LOG: database system is shut down
2026-09-29T16:23:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49455 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:23:02 dist-test-slave-s9ct krb5kdc[18209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:23:02 dist-test-slave-s9ct krb5kdc[18209](info): set up 2 sockets
Sep 29 16:23:02 dist-test-slave-s9ct krb5kdc[18209](info): commencing operation
krb5kdc: starting...
W20260929 16:23:03.266604 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:23:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:23:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:23:03.393631 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:42795
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:03.504568 18222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:03.504896 18222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:03.504978 18222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:03.509022 18222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:03.509124 18222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:03.509164 18222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:03.509219 18222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:03.509256 18222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:03.514526 18222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:03.515584 18222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:03.516611 18222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:03.525090 18228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:03.525094 18227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:03.525178 18222 server_base.cc:1034] running on GCE node
W20260929 16:23:03.525311 18230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:03.525916 18222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:03.527036 18222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:03.528234 18222 hybrid_clock.cc:648] HybridClock initialized: now 1790698983528207 us; error 48 us; skew 500 ppm
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:03.530947 18222 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:03.531777 18222 webserver.cc:460] Webserver started at http://127.4.200.126:38049/ using document root <none> and password file <none>
I20260929 16:23:03.532256 18222 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:03.532311 18222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:03.532469 18222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:03.534348 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "9e57b53cdf134b7e9032892c0df73499"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "60109c9d6b4df39bd48a8a067d16be37"
server_key_iv: "33185500065b995d32d97f09047b3731"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.534921 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "9e57b53cdf134b7e9032892c0df73499"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "60109c9d6b4df39bd48a8a067d16be37"
server_key_iv: "33185500065b995d32d97f09047b3731"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.538486 18222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:23:03.540756 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:03.541929 18222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:23:03.542057 18222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "9e57b53cdf134b7e9032892c0df73499"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "60109c9d6b4df39bd48a8a067d16be37"
server_key_iv: "33185500065b995d32d97f09047b3731"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.542160 18222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:03.574028 18222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:03.577050 18222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:03.577261 18222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:03.584601 18222 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:42795
I20260929 16:23:03.584617 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:42795 every 8 connection(s)
I20260929 16:23:03.585570 18222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:03.588218 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:03.590153 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18222
I20260929 16:23:03.590304 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:23:03.590703 4897 external_mini_cluster.cc:1428] Setting key 4a3ab6b74167d9b1fea0a02c573c941d
I20260929 16:23:03.594009 18290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.200.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.125@KRBTEST.COM" created.
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:03.610975 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.596797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42795 (local address 127.0.0.1:37010)
0929 16:23:03.597270 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.597321 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:03.598157 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:03.599018 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:03.599023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:03.599364 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:03.599697 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.599721 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.600968 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.600971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.601855 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.601866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.602594 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.602604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.602809 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.603439 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:03.603469 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:03.605669 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:03.608091 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.608099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.608107 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.608328 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.608730 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.608732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.608733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.608836 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.609583 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:03.609588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:03.609710 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 16:23:03.609851 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:23:03.610109 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":48,"thread_start_us":136,"threads_started":1}
I20260929 16:23:03.610978 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.596799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.4.200.126:42795)
0929 16:23:03.597362 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:03.597375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:03.597422 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:23:03.598499 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:03.598509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:03.598773 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:03.598987 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:03.599881 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.600845 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.601976 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.602398 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.602598 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.605837 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:03.605847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:03.605863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:03.605895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:03.607955 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.608458 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.608466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.608471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.608625 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.608930 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.608935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.608938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.609448 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:03.609582 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:03.609969 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:23:03.610183 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":441,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260929 16:23:03.613742 18290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:23:03.614666 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.614301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55310)
0929 16:23:03.614472 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.614587 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 16:23:03.614897 18239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
I20260929 16:23:03.615820 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.615566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55320)
0929 16:23:03.615682 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.615735 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:23:03.615947 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:36339: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
I20260929 16:23:03.622277 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:36339
--webserver_interface=127.4.200.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:03.665244 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } attempt: 1
I20260929 16:23:03.666271 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.665848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55328)
0929 16:23:03.666060 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.666153 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:23:03.667120 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.666870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55334)
0929 16:23:03.666992 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.667049 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:23:03.667284 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:36339: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
I20260929 16:23:03.735471 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } attempt: 2
I20260929 16:23:03.736366 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.736009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55346)
0929 16:23:03.736177 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.736261 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:23:03.737116 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.736884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55348)
0929 16:23:03.736991 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.737044 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:23:03.737270 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:36339: Network error: Client connection negotiation failed: client connection to 127.4.200.125:36339: connect: Connection refused (error 111)
W20260929 16:23:03.740203 18297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:03.740463 18297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:03.740505 18297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:03.744566 18297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:03.744619 18297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:03.744639 18297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:03.744656 18297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:03.744673 18297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:03.750375 18297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.200.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:03.751461 18297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:03.752461 18297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:03.760790 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:03.760810 18303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:03.760808 18305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:03.761168 18297 server_base.cc:1034] running on GCE node
I20260929 16:23:03.761763 18297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:03.762916 18297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:03.764070 18297 hybrid_clock.cc:648] HybridClock initialized: now 1790698983764038 us; error 48 us; skew 500 ppm
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:03.766783 18297 init.cc:377] Logged in from keytab as kudu/127.4.200.125@KRBTEST.COM (short username kudu)
I20260929 16:23:03.767573 18297 webserver.cc:460] Webserver started at http://127.4.200.125:38917/ using document root <none> and password file <none>
I20260929 16:23:03.768139 18297 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:03.768198 18297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:03.768360 18297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:03.770267 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data/instance:
uuid: "ef9402ca93db4a7fbd072412a356ae59"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "11c2d57176107b8518cc68325608a483"
server_key_iv: "d8d5fb2b9f6763649718555f94204cc0"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.770851 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal/instance:
uuid: "ef9402ca93db4a7fbd072412a356ae59"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "11c2d57176107b8518cc68325608a483"
server_key_iv: "d8d5fb2b9f6763649718555f94204cc0"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.774299 18297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:23:03.776647 18312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:03.777848 18297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:03.778000 18297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "ef9402ca93db4a7fbd072412a356ae59"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "11c2d57176107b8518cc68325608a483"
server_key_iv: "d8d5fb2b9f6763649718555f94204cc0"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.778085 18297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:03.795459 18297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:03.798907 18297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:03.799062 18297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:03.806097 18297 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.125:36339
I20260929 16:23:03.806120 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.125:36339 every 8 connection(s)
I20260929 16:23:03.807070 18297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
I20260929 16:23:03.808597 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18297
I20260929 16:23:03.808760 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-1/wal/instance
I20260929 16:23:03.809083 4897 external_mini_cluster.cc:1428] Setting key 3be8ff5b5c3a51af32e642187c228ea9
I20260929 16:23:03.809875 18365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:03.813616 18365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.200.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.124@KRBTEST.COM" created.
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:03.828444 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } attempt: 3
I20260929 16:23:03.828857 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.815663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.4.200.126:42795)
0929 16:23:03.815801 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:03.815805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:03.816811 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 16:23:03.817852 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:03.817856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:03.817921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:03.818027 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:03.819473 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.820156 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.821381 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.821623 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.821719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.824736 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:03.824743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:03.824750 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:03.824786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:03.826561 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.827079 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.827084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.827087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.827183 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.827524 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.827530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.827532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.827797 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:03.827923 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:03.828493 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 16:23:03.828646 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:23:03.829802 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.816122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42795 (local address 127.0.0.1:37024)
0929 16:23:03.816656 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.816697 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:03.817626 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:03.818265 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:03.818277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:03.818701 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:03.819315 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.819333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.820288 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.820291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.821207 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.821223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.821798 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.821804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.822038 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.822796 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:03.822813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:03.824530 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:03.826687 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.826692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.826706 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.826971 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.827289 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.827291 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.827293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.827427 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.827919 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:03.827924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:03.828208 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 16:23:03.828372 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:23:03.828753 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":160,"threads_started":1}
I20260929 16:23:03.831346 18365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
I20260929 16:23:03.837741 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:34691
--webserver_interface=127.4.200.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:03.840018 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.829033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.4.200.125:36339)
0929 16:23:03.829377 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:03.829391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:03.829414 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:23:03.829461 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:03.829465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:03.829621 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:03.829732 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:03.830828 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.831946 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.832767 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.833211 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.833292 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.835343 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:03.835352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:03.835364 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:03.835385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:03.837518 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.837973 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.837982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.837986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.838079 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.838481 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.838488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.838492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.838850 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:03.839176 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:03.839582 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 16:23:03.839777 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260929 16:23:03.840124 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.828904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55356)
0929 16:23:03.829150 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.829164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:03.829236 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:03.829758 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:03.829761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:03.829959 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:03.830672 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.830686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.832043 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.832047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.832651 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.832661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.833322 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.833326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.833431 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.833997 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:03.834019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:03.835202 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:03.837651 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.837658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.837660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.837858 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.838198 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.838200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.838202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.838334 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.839235 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:03.839239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:03.839372 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 16:23:03.839734 (+ 362us) client_negotiation.cc:240] Negotiation successful
0929 16:23:03.839909 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 16:23:03.842258 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.831855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55366)
0929 16:23:03.831946 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.831959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:03.832033 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:03.832493 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:03.832495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:03.832709 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:03.832815 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.832822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.833677 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.833682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.834316 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:03.834327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.835681 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.835688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:03.835774 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.836132 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:03.836143 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:03.837531 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:03.840742 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.840748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.840751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.841000 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.841293 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:03.841299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:03.841302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:03.841382 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:03.841748 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:03.841751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:03.841839 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 16:23:03.841940 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:23:03.842089 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 16:23:03.842574 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.831918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.4.200.125:36339)
0929 16:23:03.832309 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:03.832316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:03.832327 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:23:03.832352 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:03.832354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:03.832385 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:03.832485 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:03.832934 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.833564 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.834451 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:03.835562 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:03.835651 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:03.837976 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:03.837979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:03.837981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:03.837999 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:03.840626 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.841104 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.841107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.841109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.841211 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:03.841471 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:03.841478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:03.841479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:03.841647 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:03.841759 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:03.842005 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 16:23:03.842154 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":340,"spinlock_wait_cycles":2432,"thread_start_us":162,"threads_started":1}
I20260929 16:23:03.842515 18290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:23:03.843182 18365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:23:03.843439 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.843053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49902)
0929 16:23:03.843278 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.843344 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 16:23:03.844008 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.843650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49916)
0929 16:23:03.843890 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.843956 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260929 16:23:03.844195 18314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.844308 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.843954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49924)
0929 16:23:03.844158 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.844216 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260929 16:23:03.844492 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.845223 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.844906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49930)
0929 16:23:03.845050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.845099 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 16:23:03.845436 18365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.875676 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 1
I20260929 16:23:03.876608 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.876264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49938)
0929 16:23:03.876440 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.876511 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:23:03.877372 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.877138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49952)
0929 16:23:03.877246 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.877307 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:23:03.877511 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.894677 18365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 1
I20260929 16:23:03.895596 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.895287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49962)
0929 16:23:03.895443 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.895512 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:23:03.896382 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.896122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49970)
0929 16:23:03.896261 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.896320 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:23:03.896590 18365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.952690 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 2
I20260929 16:23:03.953595 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.953218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49982)
0929 16:23:03.953413 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.953489 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:23:03.954527 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.954190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:49992)
0929 16:23:03.954315 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.954386 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:23:03.954746 18290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
W20260929 16:23:03.954532 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:03.954918 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:03.955019 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:03.959198 18372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:03.959252 18372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:03.959275 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:03.959292 18372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:03.959342 18372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:03.964757 18372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:42795,127.4.200.125:36339,127.4.200.124:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.200.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:03.964769 18365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 2
I20260929 16:23:03.965653 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.965331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:50006)
0929 16:23:03.965496 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.965560 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:23:03.966168 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:03.966516 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:03.966209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:50022)
0929 16:23:03.966328 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:03.966383 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:23:03.966632 18365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34691: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34691: connect: Connection refused (error 111)
I20260929 16:23:03.967361 18372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:03.976174 18378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:03.976176 18379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:03.976208 18372 server_base.cc:1034] running on GCE node
W20260929 16:23:03.976387 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:03.976984 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:03.978111 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:03.979339 18372 hybrid_clock.cc:648] HybridClock initialized: now 1790698983979303 us; error 59 us; skew 500 ppm
Sep 29 16:23:03 dist-test-slave-s9ct krb5kdc[18209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:03.982455 18372 init.cc:377] Logged in from keytab as kudu/127.4.200.124@KRBTEST.COM (short username kudu)
I20260929 16:23:03.983384 18372 webserver.cc:460] Webserver started at http://127.4.200.124:40559/ using document root <none> and password file <none>
I20260929 16:23:03.983973 18372 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:03.984082 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:03.984324 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:03.986500 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data/instance:
uuid: "b60ea79568224dec8f77f5e0d63e20f3"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "352d0a641043f2046153c95d27a9148f"
server_key_iv: "9afa4cf3792f3fd4e8660fe7a416771f"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.987078 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal/instance:
uuid: "b60ea79568224dec8f77f5e0d63e20f3"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "352d0a641043f2046153c95d27a9148f"
server_key_iv: "9afa4cf3792f3fd4e8660fe7a416771f"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.990922 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:23:03.993451 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:03.994874 18372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:23:03.995021 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "b60ea79568224dec8f77f5e0d63e20f3"
format_stamp: "Formatted at 2026-09-29 16:23:03 on dist-test-slave-s9ct"
server_key: "352d0a641043f2046153c95d27a9148f"
server_key_iv: "9afa4cf3792f3fd4e8660fe7a416771f"
server_key_version: "encryptionkey@0"
I20260929 16:23:03.995153 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:04.008977 18372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:04.012302 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:04.012523 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:04.019886 18372 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.124:34691
I20260929 16:23:04.019897 18440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.124:34691 every 8 connection(s)
I20260929 16:23:04.020916 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
I20260929 16:23:04.023651 18441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:04.025658 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18372
I20260929 16:23:04.025816 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790698821455595-4897-0/minicluster-data/master-2/wal/instance
I20260929 16:23:04.026117 4897 external_mini_cluster.cc:1428] Setting key 1f07204e3a69d82e4b79e3770d833ea5
I20260929 16:23:04.028101 18441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:23:04.040464 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.030262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.4.200.126:42795)
0929 16:23:04.030435 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.030440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.030761 (+ 321us) server_negotiation.cc:407] Connection header received
0929 16:23:04.030893 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.030895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.030940 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.031010 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.031852 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.032318 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.033467 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.033698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.033814 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.036307 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.036310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.036312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.036338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.037996 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.038646 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.038651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.038653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.038736 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.039077 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.039080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.039082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.039337 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.039444 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.039959 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.040086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:23:04.041190 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.030191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42795 (local address 127.0.0.1:37032)
0929 16:23:04.030621 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:04.030671 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:04.030790 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:04.031094 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:04.031100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:04.031522 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:04.031722 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.031734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.032632 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.032636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.033342 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.033352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.033851 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.033855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.033959 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.034497 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:04.034515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:04.036134 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:04.038141 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.038151 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.038161 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.038544 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.038830 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.038835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.038837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.038975 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.039442 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:04.039448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:04.039719 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 16:23:04.039902 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 16:23:04.040211 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":46,"thread_start_us":131,"threads_started":1}
I20260929 16:23:04.041193 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.027551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.4.200.124:34691)
0929 16:23:04.027990 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.028005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.028039 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:23:04.028806 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.028825 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.029216 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.029556 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.030511 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.031970 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.032819 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.033331 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.033504 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.036688 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.036698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.036713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.036743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.038574 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.039025 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.039031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.039033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.039115 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.039425 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.039428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.039430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.039727 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.039957 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.040166 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.040304 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":300,"spinlock_wait_cycles":768,"thread_start_us":146,"threads_started":1}
I20260929 16:23:04.042478 18441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:04.052454 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.043116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.4.200.125:36339)
0929 16:23:04.043242 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.043246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.043271 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:23:04.043364 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.043367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.043412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.043485 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.044075 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.044921 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.045931 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.046327 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.046464 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.048819 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.048822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.048824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.048849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.050734 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.051192 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.051194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.051196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.051267 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.051529 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.051532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.051535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.051777 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.051863 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.052187 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.052295 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:23:04.052454 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.043030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:36339 (local address 127.0.0.1:55376)
0929 16:23:04.043190 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:04.043204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:04.043278 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:04.043623 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:04.043626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:04.043812 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:04.043939 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.043949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.045022 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.045026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.045839 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.045848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.046472 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.046475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.046556 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.047335 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:04.047346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:04.048671 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:04.050846 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.050851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.050854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.051098 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.051347 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.051349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.051350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.051424 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.051882 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:04.051888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:04.051992 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 16:23:04.052104 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:23:04.052253 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:23:04.053048 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.043568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.4.200.126:42795)
0929 16:23:04.043701 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.043705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.043768 (+ 63us) server_negotiation.cc:407] Connection header received
0929 16:23:04.043856 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.043859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.043902 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.043988 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.044793 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.045548 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.046634 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.046921 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.047042 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.049453 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.049456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.049459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.049492 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.051102 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.051727 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.051730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.051731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.051806 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.052101 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.052104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.052106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.052322 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.052549 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.052761 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.052867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:23:04.053532 18441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:23:04.053911 18290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 3
I20260929 16:23:04.057797 18365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } attempt: 3
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Sep 29 16:23:04 dist-test-slave-s9ct krb5kdc[18209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:23:04.065834 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.054255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:50046)
0929 16:23:04.054452 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:04.054463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:04.054527 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:04.055054 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:04.055060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:04.055218 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:04.055294 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.055299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.056760 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.056766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.057517 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.057526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.058172 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.058175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.058255 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.058749 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:04.058770 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:04.060559 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:04.063914 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.063920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.063925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.064212 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.064566 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.064571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.064668 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.065266 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:04.065269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:04.065377 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 16:23:04.065470 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:23:04.065633 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 16:23:04.066684 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.054075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:50044)
0929 16:23:04.054202 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:04.054216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:04.054294 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:04.054985 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:04.054988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:04.055172 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:04.055283 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.055294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.056157 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.056162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.056750 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.056760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.057336 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.057341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.057426 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.057988 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:04.058002 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:04.061151 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:04.063867 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.063870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.063871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.064333 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.064632 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.064637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.064724 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.065285 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:04.065289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:04.065395 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:23:04.065510 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:23:04.066493 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 16:23:04.066879 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.054502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.4.200.125:36339)
0929 16:23:04.054643 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.054648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.054705 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:23:04.054845 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.054848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.054893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.054978 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.055719 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.056518 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.057387 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.057737 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.057839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.062345 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.062352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.062358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.062432 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.064746 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.065261 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.065265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.065266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.065349 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.065679 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.065682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.065684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.065899 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.066111 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.066533 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.066685 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:23:04.066951 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.054428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.4.200.124:34691)
0929 16:23:04.054823 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.054832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.054843 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:23:04.054871 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.054874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.054920 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.055384 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.055451 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.056651 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.057627 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.058045 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.058471 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.060689 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.060692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.060695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.060719 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.063043 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064362 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.064365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.064367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.064452 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064788 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.064794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.064799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.065133 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.066158 (+ 1025us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.066232 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.066770 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":409,"server-negotiator.queue_time_us":301,"spinlock_wait_cycles":384,"thread_start_us":122,"threads_started":1}
I20260929 16:23:04.066692 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.054311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.4.200.124:34691)
0929 16:23:04.054687 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.054694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.054707 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:23:04.054740 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.054743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.054787 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.054859 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.055420 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.056048 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.056855 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.057214 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.057311 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.061316 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.061322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.061324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.061347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.063755 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064443 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.064446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.064448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.064523 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.064843 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.064846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.064849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.065144 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.065247 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.065759 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.066563 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":670,"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":1792}
I20260929 16:23:04.068055 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.058182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34691 (local address 127.0.0.1:50056)
0929 16:23:04.058466 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:04.058480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:04.058548 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:04.059052 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:04.059055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:04.059234 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:04.059345 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.059358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.060219 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.060225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.061124 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:04.061137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.061940 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.061960 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:04.062074 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.062703 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:04.062723 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:04.064325 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:04.066441 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.066445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.066447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.066726 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.067047 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:04.067051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:04.067052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:04.067139 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:04.067582 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:04.067585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:04.067677 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:23:04.067756 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:23:04.067868 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 16:23:04.068668 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.058489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.4.200.124:34691)
0929 16:23:04.058805 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.058816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.058830 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:23:04.058865 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.058868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.058909 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.059443 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.059542 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.060103 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.061269 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.061735 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.061947 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.064466 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.064469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.064472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.064498 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.066315 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.066835 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.066838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.066839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.066936 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.067263 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.067267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.067269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.067476 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.068294 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.068365 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.068476 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":146,"threads_started":1}
I20260929 16:23:04.070843 18441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: Bootstrap starting.
I20260929 16:23:04.070844 18290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: Bootstrap starting.
I20260929 16:23:04.071223 18365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59: Bootstrap starting.
I20260929 16:23:04.071518 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:23:04.073726 18365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:04.073732 18290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:04.073747 18441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:04.074709 18365 log.cc:824] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:04.074709 18290 log.cc:824] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:04.074712 18441 log.cc:824] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:04.077368 18441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: No bootstrap required, opened a new log
I20260929 16:23:04.077761 18365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59: No bootstrap required, opened a new log
I20260929 16:23:04.078073 18290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: No bootstrap required, opened a new log
I20260929 16:23:04.082832 18365 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.083163 18365 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:04.083269 18365 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9402ca93db4a7fbd072412a356ae59, State: Initialized, Role: FOLLOWER
I20260929 16:23:04.083406 18441 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.083657 18441 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:04.083475 18290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.083717 18441 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60ea79568224dec8f77f5e0d63e20f3, State: Initialized, Role: FOLLOWER
I20260929 16:23:04.083745 18290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:04.083824 18290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e57b53cdf134b7e9032892c0df73499, State: Initialized, Role: FOLLOWER
I20260929 16:23:04.083963 18365 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.084144 18441 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.084393 18290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.085557 18441 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:04.085557 18365 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:04.085898 18290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:04.086803 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.074233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.4.200.126:42795)
0929 16:23:04.074355 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.074359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.075837 (+ 1478us) server_negotiation.cc:407] Connection header received
0929 16:23:04.076050 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.076053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.076102 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.076154 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.077157 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.077898 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.079443 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.079769 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.079861 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.082107 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.082112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.082115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.082151 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.084035 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.084960 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.084964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.084966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.085086 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.085465 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.085469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.085471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.085731 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.085845 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.086459 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.086609 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:23:04.088007 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.089457 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.089632 18470 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:04.089787 18470 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.091969 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef9402ca93db4a7fbd072412a356ae59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e57b53cdf134b7e9032892c0df73499" is_pre_election: true
I20260929 16:23:04.092387 18264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9402ca93db4a7fbd072412a356ae59 in term 0.
I20260929 16:23:04.092692 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.092870 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.092906 18314 leader_election.cc:302] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e57b53cdf134b7e9032892c0df73499, ef9402ca93db4a7fbd072412a356ae59; no voters:
I20260929 16:23:04.093230 18458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:04.093329 18458 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:04.093389 18458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:04.094136 18414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef9402ca93db4a7fbd072412a356ae59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60ea79568224dec8f77f5e0d63e20f3" is_pre_election: true
I20260929 16:23:04.089710 18372 master_runner.cc:186] Error getting master registration for 127.4.200.126:42795: OK, instance_id { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" instance_seqno: 1790698983582509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:23:04.094470 18372 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:23:04.094513 18414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9402ca93db4a7fbd072412a356ae59 in term 0.
I20260929 16:23:04.094974 18458 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.095103 18470 leader_election.cc:288] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e57b53cdf134b7e9032892c0df73499 (127.4.200.126:42795), b60ea79568224dec8f77f5e0d63e20f3 (127.4.200.124:34691)
I20260929 16:23:04.095366 18458 leader_election.cc:288] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [CANDIDATE]: Term 1 election: Requested vote from peers 9e57b53cdf134b7e9032892c0df73499 (127.4.200.126:42795), b60ea79568224dec8f77f5e0d63e20f3 (127.4.200.124:34691)
I20260929 16:23:04.095824 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef9402ca93db4a7fbd072412a356ae59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e57b53cdf134b7e9032892c0df73499"
I20260929 16:23:04.095971 18414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef9402ca93db4a7fbd072412a356ae59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60ea79568224dec8f77f5e0d63e20f3"
I20260929 16:23:04.095983 18264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:04.096110 18414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:04.097458 18264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9402ca93db4a7fbd072412a356ae59 in term 1.
I20260929 16:23:04.097458 18414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9402ca93db4a7fbd072412a356ae59 in term 1.
I20260929 16:23:04.097802 18314 leader_election.cc:302] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60ea79568224dec8f77f5e0d63e20f3, ef9402ca93db4a7fbd072412a356ae59; no voters:
I20260929 16:23:04.086202 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.098168 18470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:04.098433 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.098294 18470 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [term 1 LEADER]: Becoming Leader. State: Replica: ef9402ca93db4a7fbd072412a356ae59, State: Running, Role: LEADER
I20260929 16:23:04.099481 18470 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } }
I20260929 16:23:04.100494 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.075536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.4.200.124:34691)
0929 16:23:04.075651 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.075656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.075902 (+ 246us) server_negotiation.cc:407] Connection header received
0929 16:23:04.076058 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.076061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.076102 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.076154 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.086352 (+ 10198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.087195 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.088417 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.088795 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.088884 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.096418 (+ 7534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.096424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.096430 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.096466 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.098345 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.098977 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.098981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.098983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.099071 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.099384 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.099387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.099388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.099643 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.099770 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.100204 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.100346 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:23:04.101886 18472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:23:04.101969 18472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:23:04.103341 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:04.075090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.4.200.125:36339)
0929 16:23:04.075208 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:04.075213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:04.075896 (+ 683us) server_negotiation.cc:407] Connection header received
0929 16:23:04.076081 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:04.076084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:04.076133 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:04.076185 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:04.087093 (+ 10908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.087867 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.090351 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:04.090739 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:04.090823 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:04.094280 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:04.094284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:04.094287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:04.094322 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:04.096371 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.097682 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.097686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.097688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.097816 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:04.102274 (+ 4458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:04.102279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:04.102281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:04.102560 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:04.102700 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:04.103037 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:23:04.103171 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 16:23:04.103441 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9402ca93db4a7fbd072412a356ae59. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.103583 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: This master's current role is: LEADER
W20260929 16:23:04.106377 18488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:23:04.106617 18488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:23:04.109974 18497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:04.113519 18497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:04.122112 18414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [term 1 FOLLOWER]: Refusing update from remote peer ef9402ca93db4a7fbd072412a356ae59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:04.123013 18458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:04.124286 18264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [term 1 FOLLOWER]: Refusing update from remote peer ef9402ca93db4a7fbd072412a356ae59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:04.126742 18470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:04.131044 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9402ca93db4a7fbd072412a356ae59. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.131207 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.135681 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9402ca93db4a7fbd072412a356ae59. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.135846 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.141474 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.141650 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:04.143103 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.143246 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.143532 18500 mvcc.cc:205] Tried to move back new op lower bound from 7334703038956515328 to 7334703038875070464. Current Snapshot: MvccSnapshot[applied={T|T < 7334703038956515328}]
I20260929 16:23:04.143707 18501 mvcc.cc:205] Tried to move back new op lower bound from 7334703038956515328 to 7334703038875070464. Current Snapshot: MvccSnapshot[applied={T|T < 7334703038956515328}]
I20260929 16:23:04.147949 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.148180 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:04.148782 18497 catalog_manager.cc:1347] Generated new cluster ID: 29a5fab5c6984312a150a2ab3c1cd01d
I20260929 16:23:04.148849 18497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:04.150722 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef9402ca93db4a7fbd072412a356ae59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e57b53cdf134b7e9032892c0df73499" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42795 } } peers { permanent_uuid: "ef9402ca93db4a7fbd072412a356ae59" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 36339 } } peers { permanent_uuid: "b60ea79568224dec8f77f5e0d63e20f3" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34691 } } }
I20260929 16:23:04.150862 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:23:04.151520 18503 mvcc.cc:205] Tried to move back new op lower bound from 7334703038956515328 to 7334703038875070464. Current Snapshot: MvccSnapshot[applied={T|T < 7334703038956515328}]
I20260929 16:23:04.167465 18497 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:23:04.169234 18497 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:04.189054 18497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9402ca93db4a7fbd072412a356ae59: Generated new TSK 0
I20260929 16:23:04.190102 18497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:23:05.103330 18472 catalog_manager.cc:1259] Loaded cluster ID: 29a5fab5c6984312a150a2ab3c1cd01d
I20260929 16:23:05.103422 18472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: loading cluster ID for follower catalog manager: success
I20260929 16:23:05.105036 18472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: acquiring CA information for follower catalog manager: success
I20260929 16:23:05.106182 18472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b60ea79568224dec8f77f5e0d63e20f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:23:05.107695 18488 catalog_manager.cc:1259] Loaded cluster ID: 29a5fab5c6984312a150a2ab3c1cd01d
I20260929 16:23:05.107762 18488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: loading cluster ID for follower catalog manager: success
I20260929 16:23:05.109256 18488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: acquiring CA information for follower catalog manager: success
I20260929 16:23:05.110338 18488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e57b53cdf134b7e9032892c0df73499: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:23:05.260143 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18222
I20260929 16:23:05.266285 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18297
I20260929 16:23:05.272275 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18372
2026-09-29T16:23:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3054 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:23:05 dist-test-slave-s9ct krb5kdc[18515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:23:05 dist-test-slave-s9ct krb5kdc[18515](info): set up 2 sockets
Sep 29 16:23:05 dist-test-slave-s9ct krb5kdc[18515](info): commencing operation
krb5kdc: starting...
W20260929 16:23:06.307013 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:23:06.430099 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33365
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:06.543290 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:06.543603 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:06.543694 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:06.547847 18528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:06.547936 18528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:06.547974 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:06.548034 18528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:06.548072 18528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:06.553428 18528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:06.554711 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:06.555797 18528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:06.564539 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:06.564546 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:06.564605 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:06.564999 18528 server_base.cc:1034] running on GCE node
I20260929 16:23:06.565618 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:06.566787 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:06.567976 18528 hybrid_clock.cc:648] HybridClock initialized: now 1790698986567962 us; error 28 us; skew 500 ppm
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698986, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:06.570803 18528 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:06.571682 18528 webserver.cc:460] Webserver started at http://127.4.200.126:44175/ using document root <none> and password file <none>
I20260929 16:23:06.572248 18528 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:06.572352 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:06.572589 18528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:06.574719 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "5626d70607b349c6bd166d5bc37183e4"
format_stamp: "Formatted at 2026-09-29 16:23:06 on dist-test-slave-s9ct"
server_key: "8e4add9de6a5fd6f75165bc556384d7f"
server_key_iv: "6a4e5a5c8a503e43aa41c19881d80ed4"
server_key_version: "encryptionkey@0"
I20260929 16:23:06.575291 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "5626d70607b349c6bd166d5bc37183e4"
format_stamp: "Formatted at 2026-09-29 16:23:06 on dist-test-slave-s9ct"
server_key: "8e4add9de6a5fd6f75165bc556384d7f"
server_key_iv: "6a4e5a5c8a503e43aa41c19881d80ed4"
server_key_version: "encryptionkey@0"
I20260929 16:23:06.579082 18528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:23:06.581458 18543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:06.582775 18528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 16:23:06.582914 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "5626d70607b349c6bd166d5bc37183e4"
format_stamp: "Formatted at 2026-09-29 16:23:06 on dist-test-slave-s9ct"
server_key: "8e4add9de6a5fd6f75165bc556384d7f"
server_key_iv: "6a4e5a5c8a503e43aa41c19881d80ed4"
server_key_version: "encryptionkey@0"
I20260929 16:23:06.583026 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:06.606614 18528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:06.609957 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:06.610163 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:06.617182 18528 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:33365
I20260929 16:23:06.617261 18595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:33365 every 8 connection(s)
I20260929 16:23:06.618155 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:06.620961 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:06.626276 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4: Bootstrap starting.
I20260929 16:23:06.626681 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18528
I20260929 16:23:06.626835 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:23:06.627157 4897 external_mini_cluster.cc:1428] Setting key a460f7b7cc8fd7455f3c71ef7c126755
I20260929 16:23:06.628976 18596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:06.629678 18596 log.cc:824] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:06.631850 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4: No bootstrap required, opened a new log
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:06.636456 18596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:23:06.636667 18596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:06.636737 18596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5626d70607b349c6bd166d5bc37183e4, State: Initialized, Role: FOLLOWER
I20260929 16:23:06.637212 18596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:23:06.637394 18596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:06.637491 18596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:06.637601 18596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:06.638856 18596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:23:06.639236 18596 leader_election.cc:302] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5626d70607b349c6bd166d5bc37183e4; no voters:
I20260929 16:23:06.639568 18596 leader_election.cc:288] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:23:06.639662 18601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:06.639905 18601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5626d70607b349c6bd166d5bc37183e4, State: Running, Role: LEADER
I20260929 16:23:06.640214 18601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:23:06.640879 18596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:06.641440 18603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5626d70607b349c6bd166d5bc37183e4. Latest consensus state: current_term: 1 leader_uuid: "5626d70607b349c6bd166d5bc37183e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } } }
I20260929 16:23:06.641422 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5626d70607b349c6bd166d5bc37183e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5626d70607b349c6bd166d5bc37183e4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } } }
I20260929 16:23:06.641564 18603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:06.641564 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:06.642695 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:06.628427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.4.200.126:33365)
0929 16:23:06.628842 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:06.628856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:06.628890 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:23:06.629558 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:06.629575 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:06.629944 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:06.630322 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:06.631179 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:06.632206 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:06.633345 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:06.633784 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:06.633904 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:06.636832 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:06.636838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:06.636860 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:06.636887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:06.639038 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:06.639499 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:06.639505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:06.639511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:06.639659 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:06.639977 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:06.639981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:06.639982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:06.640488 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:06.640678 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:06.641339 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 16:23:06.641657 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
I20260929 16:23:06.641964 18607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:06.645574 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:23:06.646118 18607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:06.651731 18607 catalog_manager.cc:1347] Generated new cluster ID: 8327b60209a7455a887bf5791ba1b573
I20260929 16:23:06.651811 18607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:06.682883 18607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:23:06.683892 18607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:06.688318 18607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5626d70607b349c6bd166d5bc37183e4: Generated new TSK 0
I20260929 16:23:06.689100 18607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:23:06.701375 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18528
2026-09-29T16:23:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1427 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18623](info): set up 2 sockets
Sep 29 16:23:06 dist-test-slave-s9ct krb5kdc[18623](info): commencing operation
krb5kdc: starting...
W20260929 16:23:07.734828 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:23:07 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:23:07.861856 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:07.973403 18636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:07.973738 18636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:07.973820 18636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:07.977849 18636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:07.977944 18636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:07.977985 18636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:07.978048 18636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:07.978087 18636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:07.986982 18636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:07.988332 18636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:07.989382 18636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:07.998333 18641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:07.998492 18642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:07.998335 18644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:07.998586 18636 server_base.cc:1034] running on GCE node
I20260929 16:23:07.999322 18636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:08.000550 18636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:08.001734 18636 hybrid_clock.cc:648] HybridClock initialized: now 1790698988001720 us; error 33 us; skew 500 ppm
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:08.011197 18636 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:08.012202 18636 webserver.cc:460] Webserver started at http://127.4.200.126:40999/ using document root <none> and password file <none>
I20260929 16:23:08.012794 18636 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:08.012862 18636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:08.013089 18636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:08.015266 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "ca098de226bc43b3beabb054230000ed"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "8646cdeeb2b490f6871248e5098c63de"
server_key_iv: "a8803703cfc8ec179c6ab1c1815f29ea"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.015853 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "ca098de226bc43b3beabb054230000ed"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "8646cdeeb2b490f6871248e5098c63de"
server_key_iv: "a8803703cfc8ec179c6ab1c1815f29ea"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.019801 18636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:23:08.022235 18651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.023553 18636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:23:08.023701 18636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ca098de226bc43b3beabb054230000ed"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "8646cdeeb2b490f6871248e5098c63de"
server_key_iv: "a8803703cfc8ec179c6ab1c1815f29ea"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.023815 18636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:08.048488 18636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:08.051709 18636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:08.051916 18636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:08.059443 18636 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:39475
I20260929 16:23:08.059800 18703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:39475 every 8 connection(s)
I20260929 16:23:08.060464 18636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:08.063567 18704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:08.068621 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18636
I20260929 16:23:08.068797 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:23:08.069077 4897 external_mini_cluster.cc:1428] Setting key ac6ce7c4989ebadcad3862cf23a649f4
I20260929 16:23:08.069700 18704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap starting.
I20260929 16:23:08.072736 18704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:08.073757 18704 log.cc:824] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:08.076153 18704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: No bootstrap required, opened a new log
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:08.081130 18704 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:08.081509 18704 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:08.081599 18704 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca098de226bc43b3beabb054230000ed, State: Initialized, Role: FOLLOWER
I20260929 16:23:08.082226 18704 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:08.082466 18704 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:08.082559 18704 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:08.082712 18704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:08.084182 18704 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:08.084617 18704 leader_election.cc:302] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca098de226bc43b3beabb054230000ed; no voters:
I20260929 16:23:08.085105 18704 leader_election.cc:288] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:23:08.085203 18709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:08.085417 18709 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 LEADER]: Becoming Leader. State: Replica: ca098de226bc43b3beabb054230000ed, State: Running, Role: LEADER
I20260929 16:23:08.085796 18709 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:08.086241 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.070666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.4.200.126:39475)
0929 16:23:08.071157 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.071181 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.071231 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:23:08.072182 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.072207 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.072677 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.073110 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:08.073757 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.075397 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.076328 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.077095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.077288 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.079895 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:08.079904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:08.079928 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:08.079971 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:08.082297 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.082928 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.082939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.082949 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.083121 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.083402 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.083406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.083408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.084043 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:08.084280 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.084877 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.085140 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":124,"threads_started":1}
I20260929 16:23:08.086671 18704 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:08.087670 18710 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca098de226bc43b3beabb054230000ed. Latest consensus state: current_term: 1 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:08.087769 18711 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:08.087859 18711 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:08.087816 18710 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:08.089473 18718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:08.092427 18718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:08.097970 18718 catalog_manager.cc:1347] Generated new cluster ID: 750eab6f9e944aa8aab91499a92d26a2
I20260929 16:23:08.098052 18718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:08.113476 18718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:23:08.114596 18718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:08.123672 18718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Generated new TSK 0
I20260929 16:23:08.124421 18718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:23:08.158089 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:08.271186 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:08.271445 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:08.271488 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:08.275431 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:08.275482 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:08.275539 18730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:23:08.280622 18730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:08.281586 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:08.282569 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:08.292176 18735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.292191 18736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.292191 18738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:08.292379 18730 server_base.cc:1034] running on GCE node
I20260929 16:23:08.292764 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:08.293401 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:08.294582 18730 hybrid_clock.cc:648] HybridClock initialized: now 1790698988294567 us; error 41 us; skew 500 ppm
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:08.297379 18730 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:23:08.298245 18730 webserver.cc:460] Webserver started at http://127.4.200.65:44547/ using document root <none> and password file <none>
I20260929 16:23:08.298877 18730 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:08.298969 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:08.299199 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:08.301241 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "826b6a4a29c641aca711b18274172d01"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "a6ea001f4711c9e8fc52a0a6add07542"
server_key_iv: "4a04f32a9119830dcd90564ae5502d3c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.301822 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "826b6a4a29c641aca711b18274172d01"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "a6ea001f4711c9e8fc52a0a6add07542"
server_key_iv: "4a04f32a9119830dcd90564ae5502d3c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.305642 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:23:08.308019 18745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.309226 18730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:23:08.309373 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "826b6a4a29c641aca711b18274172d01"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "a6ea001f4711c9e8fc52a0a6add07542"
server_key_iv: "4a04f32a9119830dcd90564ae5502d3c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.309484 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:08.328325 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:08.331602 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:08.331811 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:08.332402 18730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:08.333214 18730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:08.333283 18730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.333379 18730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:08.333431 18730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.342249 18730 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44275
I20260929 16:23:08.342301 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44275 every 8 connection(s)
I20260929 16:23:08.343166 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:23:08.344796 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18730
I20260929 16:23:08.344986 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:23:08.345281 4897 external_mini_cluster.cc:1428] Setting key 8cc02a356d3be3c2d6788a8c87fa5f68
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:23:08.358078 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.344916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:53673 (local address 127.4.200.126:39475)
0929 16:23:08.345057 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.345061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.346502 (+ 1441us) server_negotiation.cc:407] Connection header received
0929 16:23:08.347411 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.347418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.347480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.347576 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:08.348965 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.349500 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.351308 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.351529 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.351612 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.354501 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:08.354504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:08.354507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:08.354531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:08.356007 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.356546 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.356550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.356552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.356632 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.356978 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.356981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.356983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.357164 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:08.357270 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.357765 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.357895 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:23:08.359230 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.345299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:53673)
0929 16:23:08.346308 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.346360 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.347212 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.347742 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.347749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.348092 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:08.348810 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.348829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.349632 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.349638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.351183 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.351195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.351697 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.351704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.351960 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.352594 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:08.352621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:08.354295 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:08.356147 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.356153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.356168 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.356436 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.356735 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.356738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.356740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.356878 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.357269 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:08.357276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:08.357529 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.357711 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.358139 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":659,"threads_started":1}
I20260929 16:23:08.360826 18859 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:08.361078 18859 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:08.361577 18859 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:08.363202 18668 ts_manager.cc:194] Registered new tserver with Master: 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275)
I20260929 16:23:08.364338 18668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:53673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:23:08.373786 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:08.484342 18864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:08.484603 18864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:08.484647 18864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:08.488962 18864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:08.489019 18864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:08.489078 18864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:23:08.494199 18864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:08.495555 18864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:08.496582 18864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:08.507833 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.507833 18869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.508571 18870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:08.508857 18864 server_base.cc:1034] running on GCE node
I20260929 16:23:08.509456 18864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:08.510171 18864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:08.513453 18864 hybrid_clock.cc:648] HybridClock initialized: now 1790698988513400 us; error 63 us; skew 500 ppm
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:08.516333 18864 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:23:08.517190 18864 webserver.cc:460] Webserver started at http://127.4.200.66:44199/ using document root <none> and password file <none>
I20260929 16:23:08.517732 18864 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:08.517789 18864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:08.517961 18864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:08.520113 18864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "b49f71eee20b41399f9cf8ada7807abc"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "709e731ceb1341f7cc338cb3fa21a9c4"
server_key_iv: "3a55f19c5c1f6593edfda3a2f2823a1c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.520689 18864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "b49f71eee20b41399f9cf8ada7807abc"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "709e731ceb1341f7cc338cb3fa21a9c4"
server_key_iv: "3a55f19c5c1f6593edfda3a2f2823a1c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.524466 18864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:23:08.526950 18879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.528292 18864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:23:08.528434 18864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "b49f71eee20b41399f9cf8ada7807abc"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "709e731ceb1341f7cc338cb3fa21a9c4"
server_key_iv: "3a55f19c5c1f6593edfda3a2f2823a1c"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.528549 18864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:08.543473 18864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:08.547150 18864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:08.547333 18864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:08.547935 18864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:08.548785 18864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:08.548838 18864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.548883 18864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:08.548908 18864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.558882 18864 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:46037
I20260929 16:23:08.558899 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:46037 every 8 connection(s)
I20260929 16:23:08.559872 18864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:23:08.560194 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18864
I20260929 16:23:08.560357 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:23:08.560674 4897 external_mini_cluster.cc:1428] Setting key 5ab45936c1396bdde619a699d00b83ee
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:08.578099 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.561916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:35609 (local address 127.4.200.126:39475)
0929 16:23:08.562071 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.562076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.563150 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 16:23:08.564952 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.564955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.565002 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.565075 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:08.566303 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.566909 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.568071 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.568316 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.568454 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.573765 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:08.573772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:08.573779 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:08.573819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:08.575574 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.576263 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.576266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.576268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.576351 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.576782 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.576784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.576785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.577033 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:08.577129 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.577852 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.577969 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:23:08.579511 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.562499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:35609)
0929 16:23:08.562971 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.563026 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.564682 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.565302 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.565309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.565670 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:08.566150 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.566164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.567041 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.567045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.567932 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.567941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.568499 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.568503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.568684 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.569382 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:08.569399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:08.573558 (+ 4159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:08.575712 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.575722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.575743 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.576141 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.576465 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.576471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.576476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.576682 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.577168 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:08.577178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:08.577567 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.577809 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.578228 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
I20260929 16:23:08.580936 18993 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:08.581221 18993 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:08.581740 18993 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:08.582901 18668 ts_manager.cc:194] Registered new tserver with Master: b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037)
I20260929 16:23:08.583614 18668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:35609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:23:08.589432 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:08.705219 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:08.705502 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:08.705546 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:08.709697 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:08.709774 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:08.709849 18998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:23:08.715569 18998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:08.716641 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:08.717624 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:08.729501 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.729480 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:08.729739 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:08.730022 18998 server_base.cc:1034] running on GCE node
I20260929 16:23:08.730559 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:08.731266 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:08.732492 18998 hybrid_clock.cc:648] HybridClock initialized: now 1790698988732467 us; error 84 us; skew 500 ppm
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:08.735357 18998 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:23:08.736246 18998 webserver.cc:460] Webserver started at http://127.4.200.67:35429/ using document root <none> and password file <none>
I20260929 16:23:08.736816 18998 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:08.736907 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:08.737133 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:08.739322 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "c0304336845a561559f9a780536bc227"
server_key_iv: "4cbd663ef3ce29153f2dc3cbd18bf071"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.739894 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "c0304336845a561559f9a780536bc227"
server_key_iv: "4cbd663ef3ce29153f2dc3cbd18bf071"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.743745 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:23:08.746342 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.747786 18998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:23:08.747941 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "c0304336845a561559f9a780536bc227"
server_key_iv: "4cbd663ef3ce29153f2dc3cbd18bf071"
server_key_version: "encryptionkey@0"
I20260929 16:23:08.748060 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:08.771178 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:08.774719 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:08.774932 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:08.775555 18998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:08.776433 18998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:08.776501 18998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.776574 18998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:08.776615 18998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:08.785503 18998 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:36111
I20260929 16:23:08.785519 19126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:36111 every 8 connection(s)
I20260929 16:23:08.786553 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
Sep 29 16:23:08 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:08.796342 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 18998
I20260929 16:23:08.796478 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:23:08.796779 4897 external_mini_cluster.cc:1428] Setting key ea1a691cae707c3f73d38daa7941e80d
I20260929 16:23:08.800602 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.788474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:43215 (local address 127.4.200.126:39475)
0929 16:23:08.788615 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.788619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.789488 (+ 869us) server_negotiation.cc:407] Connection header received
0929 16:23:08.790832 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.790835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.790884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.790958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:08.792060 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.792583 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.793689 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.793889 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.793963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.796680 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:08.796684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:08.796686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:08.796712 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:08.798524 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.799190 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.799193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.799195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.799286 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.799613 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.799617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.799618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.799810 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:08.799889 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.800372 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.800482 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:23:08.801448 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.788875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:43215)
0929 16:23:08.789323 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.789366 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.790627 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.791096 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.791102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.791458 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:08.791928 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.791941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.792695 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.792698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.793580 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.793589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.794017 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.794020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.794188 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.794848 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:08.794863 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:08.796485 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:08.798682 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.798688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.798706 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.799109 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.799373 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:08.799375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:08.799376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:08.799514 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:08.799898 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:08.799904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:08.800145 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.800305 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.800576 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260929 16:23:08.802726 19127 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:08.802985 19127 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:08.803516 19127 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:08.804630 18668 ts_manager.cc:194] Registered new tserver with Master: 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
I20260929 16:23:08.805192 18668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:43215
I20260929 16:23:08.811110 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:23:08.820827 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.813132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.4.200.126:39475)
0929 16:23:08.813272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.813277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.813434 (+ 157us) server_negotiation.cc:407] Connection header received
0929 16:23:08.813593 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.813596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.813642 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.813710 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:08.814498 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.815021 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.815856 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.816065 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.816139 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.817534 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:08.817538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:08.817540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:08.817567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:08.819218 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.819634 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.819637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.819639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.819717 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:08.819946 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:08.819948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:08.819949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:08.820171 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:08.820275 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.820488 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.820591 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:23:08.824040 18668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:23:08.827292 18668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:23:08.842595 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.837242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46037 (local address 127.0.0.1:58976)
0929 16:23:08.838029 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.838041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.838119 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.838735 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.838738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.838757 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:08.838842 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.838848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.840116 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.840129 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.841217 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.841227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.841990 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.841993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.842093 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842128 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.842268 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.842393 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260929 16:23:08.842772 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.836556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.4.200.67:36111)
0929 16:23:08.837210 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.837222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.837424 (+ 202us) server_negotiation.cc:407] Connection header received
0929 16:23:08.837715 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.837721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.837921 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.838059 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:08.838526 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.839409 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.841088 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.841918 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.842028 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842167 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.842510 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.842609 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":181,"threads_started":1}
I20260929 16:23:08.842943 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.836455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44275 (local address 127.0.0.1:54190)
0929 16:23:08.837483 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.837496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.837556 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.838096 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.838102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.838143 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:08.838358 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.838366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.839806 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.839809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.840955 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.840966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.842276 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.842280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.842368 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842413 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.842484 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.842542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
I20260929 16:23:08.842955 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.837657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.4.200.66:46037)
0929 16:23:08.838282 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.838296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.838317 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:23:08.838387 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.838418 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.838585 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.838740 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:08.838960 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.839984 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.841342 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.841864 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.841992 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842144 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.842506 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.842609 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":117,"threads_started":1}
I20260929 16:23:08.843191 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.836442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:36111 (local address 127.0.0.1:59962)
0929 16:23:08.837271 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.837340 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.837537 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.838242 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.838247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.838266 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:08.838431 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.838441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.839547 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.839551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.840966 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.840972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.842234 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.842238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:08.842323 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842345 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:23:08.842975 (+ 630us) client_negotiation.cc:240] Negotiation successful
0929 16:23:08.843063 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":62,"thread_start_us":89,"threads_started":1}
I20260929 16:23:08.843524 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.836997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.4.200.65:44275)
0929 16:23:08.837574 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.837590 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.837612 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:23:08.837692 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.837699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.837918 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.838346 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:08.838513 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.839665 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.841378 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.842154 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.842260 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:08.842382 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:08.843218 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 16:23:08.843346 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":116,"threads_started":1}
I20260929 16:23:08.845808 19061 tablet_service.cc:1467] Processing CreateTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 (DEFAULT_TABLE table=test-table [id=ec9faf0326fb4f0cb9dbc5e1be205565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:08.845927 18927 tablet_service.cc:1467] Processing CreateTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 (DEFAULT_TABLE table=test-table [id=ec9faf0326fb4f0cb9dbc5e1be205565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:08.845927 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 (DEFAULT_TABLE table=test-table [id=ec9faf0326fb4f0cb9dbc5e1be205565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:08.846848 19061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c26c3ff9ae34ee1b369703c319ab650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:08.846938 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c26c3ff9ae34ee1b369703c319ab650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:08.846938 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c26c3ff9ae34ee1b369703c319ab650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:08.852756 19144 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Bootstrap starting.
I20260929 16:23:08.854606 19145 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Bootstrap starting.
I20260929 16:23:08.854663 19146 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Bootstrap starting.
I20260929 16:23:08.855517 19144 tablet_bootstrap.cc:654] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:08.856317 19145 tablet_bootstrap.cc:654] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:08.856802 19144 log.cc:824] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:08.857156 19146 tablet_bootstrap.cc:654] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:08.857378 19145 log.cc:824] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:08.857978 19146 log.cc:824] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:08.859105 19144 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: No bootstrap required, opened a new log
I20260929 16:23:08.859318 19144 ts_tablet_manager.cc:1397] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:23:08.859685 19145 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: No bootstrap required, opened a new log
I20260929 16:23:08.859881 19145 ts_tablet_manager.cc:1397] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:23:08.860126 19146 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: No bootstrap required, opened a new log
I20260929 16:23:08.860303 19146 ts_tablet_manager.cc:1397] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:08.862797 19145 raft_consensus.cc:348] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.863057 19145 raft_consensus.cc:374] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:08.863109 19145 raft_consensus.cc:729] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826b6a4a29c641aca711b18274172d01, State: Initialized, Role: FOLLOWER
I20260929 16:23:08.863349 19144 raft_consensus.cc:348] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.863621 19144 raft_consensus.cc:374] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:08.863685 19144 raft_consensus.cc:729] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec09ed46c5a4dc4922db6086a21de6b, State: Initialized, Role: FOLLOWER
I20260929 16:23:08.863577 19145 consensus_queue.cc:260] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.864120 19144 consensus_queue.cc:260] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.864104 19146 raft_consensus.cc:348] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.864363 19146 raft_consensus.cc:374] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:08.864423 19146 raft_consensus.cc:729] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49f71eee20b41399f9cf8ada7807abc, State: Initialized, Role: FOLLOWER
I20260929 16:23:08.864652 19145 ts_tablet_manager.cc:1428] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:23:08.864871 19146 consensus_queue.cc:260] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.865032 18859 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
I20260929 16:23:08.865104 19144 ts_tablet_manager.cc:1428] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:23:08.865447 19127 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
I20260929 16:23:08.866704 19146 ts_tablet_manager.cc:1428] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:23:08.867051 18993 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
I20260929 16:23:08.995633 19151 raft_consensus.cc:482] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:08.995857 19151 raft_consensus.cc:504] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:08.997100 19151 leader_election.cc:288] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275), b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037)
I20260929 16:23:09.002141 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.997496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:41639 (local address 127.4.200.65:44275)
0929 16:23:08.997643 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.997648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.997671 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:23:08.997735 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.997737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.997778 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.997848 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:08.998208 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.999067 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.000856 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.001491 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.001632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.001678 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.001928 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.002005 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:23:09.002143 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.997285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44275 (local address 127.4.200.67:41639)
0929 16:23:08.997457 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.997473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.997569 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.997898 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.997903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.997926 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:08.998064 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.998070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.999200 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.999205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.000678 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.000690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.001619 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.001622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.001758 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.001824 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.001893 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.001981 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":217}
I20260929 16:23:09.002182 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.997535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:59545 (local address 127.4.200.66:46037)
0929 16:23:08.997656 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:08.997661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:08.997945 (+ 284us) server_negotiation.cc:407] Connection header received
0929 16:23:08.998010 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:08.998012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:08.998051 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:08.998114 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:08.998372 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.999106 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.000865 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.001491 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.001623 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.001678 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.001928 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.002005 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:23:09.002152 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:08.997404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46037 (local address 127.4.200.67:59545)
0929 16:23:08.997807 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:08.997843 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:08.997940 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:08.998144 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:08.998150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:08.998171 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:08.998269 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:08.998273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:08.999217 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:08.999220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.000698 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.000711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.001619 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.001622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.001751 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.001812 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.001893 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.001981 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":82,"spinlock_wait_cycles":3328,"thread_start_us":133,"threads_started":1}
I20260929 16:23:09.002933 18947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c26c3ff9ae34ee1b369703c319ab650" candidate_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49f71eee20b41399f9cf8ada7807abc" is_pre_election: true
I20260929 16:23:09.003003 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c26c3ff9ae34ee1b369703c319ab650" candidate_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826b6a4a29c641aca711b18274172d01" is_pre_election: true
I20260929 16:23:09.003268 18947 raft_consensus.cc:2454] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec09ed46c5a4dc4922db6086a21de6b in term 0.
I20260929 16:23:09.003298 18813 raft_consensus.cc:2454] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec09ed46c5a4dc4922db6086a21de6b in term 0.
I20260929 16:23:09.003813 19017 leader_election.cc:302] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ec09ed46c5a4dc4922db6086a21de6b, 826b6a4a29c641aca711b18274172d01, b49f71eee20b41399f9cf8ada7807abc; no voters:
I20260929 16:23:09.004109 19151 raft_consensus.cc:2781] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:09.004185 19151 raft_consensus.cc:482] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:09.004215 19151 raft_consensus.cc:3037] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:09.005203 19151 raft_consensus.cc:504] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:09.005560 19151 leader_election.cc:288] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [CANDIDATE]: Term 1 election: Requested vote from peers 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275), b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037)
I20260929 16:23:09.005894 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c26c3ff9ae34ee1b369703c319ab650" candidate_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826b6a4a29c641aca711b18274172d01"
I20260929 16:23:09.005968 18947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c26c3ff9ae34ee1b369703c319ab650" candidate_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49f71eee20b41399f9cf8ada7807abc"
I20260929 16:23:09.006044 18813 raft_consensus.cc:3037] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:09.006136 18947 raft_consensus.cc:3037] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:09.007184 18813 raft_consensus.cc:2454] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec09ed46c5a4dc4922db6086a21de6b in term 1.
I20260929 16:23:09.007221 18947 raft_consensus.cc:2454] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec09ed46c5a4dc4922db6086a21de6b in term 1.
I20260929 16:23:09.007489 19016 leader_election.cc:302] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ec09ed46c5a4dc4922db6086a21de6b, 826b6a4a29c641aca711b18274172d01; no voters:
I20260929 16:23:09.007658 19151 raft_consensus.cc:2781] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:09.007937 19151 raft_consensus.cc:686] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 LEADER]: Becoming Leader. State: Replica: 6ec09ed46c5a4dc4922db6086a21de6b, State: Running, Role: LEADER
I20260929 16:23:09.008296 19151 consensus_queue.cc:237] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:09.011348 18667 catalog_manager.cc:5495] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67). New cstate: current_term: 1 leader_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } health_report { overall_health: UNKNOWN } } }
W20260929 16:23:09.037441 19128 tablet.cc:2369] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:09.037997 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.034701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.4.200.67:36111)
0929 16:23:09.034835 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.034838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.034852 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:23:09.034960 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.034962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.035014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.035099 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:09.035300 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.035785 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.036558 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.036754 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.036826 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.037077 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:09.037594 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:09.037706 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.037795 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.037851 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:23:09.043692 18947 raft_consensus.cc:1261] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Refusing update from remote peer 6ec09ed46c5a4dc4922db6086a21de6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:09.043692 18813 raft_consensus.cc:1261] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Refusing update from remote peer 6ec09ed46c5a4dc4922db6086a21de6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:09.044561 19154 consensus_queue.cc:1035] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:09.044762 19151 consensus_queue.cc:1035] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:09.052881 19158 mvcc.cc:205] Tried to move back new op lower bound from 7334703059115159552 to 7334703058980454400. Current Snapshot: MvccSnapshot[applied={T|T < 7334703059115159552}]
I20260929 16:23:09.053387 19157 mvcc.cc:205] Tried to move back new op lower bound from 7334703059115159552 to 7334703058980454400. Current Snapshot: MvccSnapshot[applied={T|T < 7334703059115159552}]
I20260929 16:23:09.061595 18667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:09.061745 18667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:09.064158 18667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Sending DeleteTablet for 3 replicas of tablet 0c26c3ff9ae34ee1b369703c319ab650
I20260929 16:23:09.065019 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:09 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:58976
I20260929 16:23:09.065100 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:09 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:54190
I20260929 16:23:09.065719 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18730
I20260929 16:23:09.065840 19061 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:09 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:59962
I20260929 16:23:09.067467 19166 tablet_replica.cc:330] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: stopping tablet replica
I20260929 16:23:09.067518 19168 tablet_replica.cc:330] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: stopping tablet replica
I20260929 16:23:09.067814 19166 raft_consensus.cc:2229] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:09.067816 19168 raft_consensus.cc:2229] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:23:09.068156 19166 raft_consensus.cc:2258] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:09.068184 19168 raft_consensus.cc:2258] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:09.069649 19166 ts_tablet_manager.cc:1905] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:09.072345 19166 ts_tablet_manager.cc:1918] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:09.072494 19166 log.cc:1196] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/0c26c3ff9ae34ee1b369703c319ab650
I20260929 16:23:09.072844 19166 ts_tablet_manager.cc:1939] T 0c26c3ff9ae34ee1b369703c319ab650 P b49f71eee20b41399f9cf8ada7807abc: Deleting consensus metadata
I20260929 16:23:09.074100 18655 catalog_manager.cc:4841] TS b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): tablet 0c26c3ff9ae34ee1b369703c319ab650 (table test-table [id=ec9faf0326fb4f0cb9dbc5e1be205565]) successfully deleted
W20260929 16:23:09.074334 18654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:44275 (error 108)
I20260929 16:23:09.075158 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18864
I20260929 16:23:09.075359 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.075045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44275 (local address 127.0.0.1:54196)
0929 16:23:09.075179 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.075261 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 16:23:09.077039 18654 catalog_manager.cc:4584] TS 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c26c3ff9ae34ee1b369703c319ab650: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44275: connect: Connection refused (error 111)
I20260929 16:23:09.078540 19168 ts_tablet_manager.cc:1905] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:09.081916 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18998
I20260929 16:23:09.087014 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 18636
I20260929 16:23:09.093652 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--webserver_interface=127.4.200.126
--webserver_port=40999
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:09.201618 19170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:09.201925 19170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:09.202010 19170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:09.207018 19170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:09.207150 19170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:09.207201 19170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:09.207262 19170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:09.207302 19170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:09.212775 19170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=40999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:09.213806 19170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:09.214829 19170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:09.227780 19176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.227775 19179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.227771 19177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:09.228124 19170 server_base.cc:1034] running on GCE node
I20260929 16:23:09.228693 19170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:09.229795 19170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:09.231006 19170 hybrid_clock.cc:648] HybridClock initialized: now 1790698989230975 us; error 53 us; skew 500 ppm
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:09.233785 19170 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:09.234791 19170 webserver.cc:460] Webserver started at http://127.4.200.126:40999/ using document root <none> and password file <none>
I20260929 16:23:09.235381 19170 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:09.235493 19170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:09.239316 19170 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:23:09.241389 19186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.242619 19170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:23:09.242763 19170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ca098de226bc43b3beabb054230000ed"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "8646cdeeb2b490f6871248e5098c63de"
server_key_iv: "a8803703cfc8ec179c6ab1c1815f29ea"
server_key_version: "encryptionkey@0"
I20260929 16:23:09.243343 19170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:09.254917 19170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:09.257633 19170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:09.257838 19170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:09.265024 19170 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:39475
I20260929 16:23:09.265323 19238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:39475 every 8 connection(s)
I20260929 16:23:09.266120 19170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:09.269618 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19170
I20260929 16:23:09.270100 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:44275
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=44547
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:09.270843 19239 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:23:09.272868 19239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap starting.
I20260929 16:23:09.289835 19239 log.cc:824] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:09.298444 19239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:09.298846 19239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap complete.
I20260929 16:23:09.302955 19239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:09.303349 19239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca098de226bc43b3beabb054230000ed, State: Initialized, Role: FOLLOWER
I20260929 16:23:09.303885 19239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:09.304004 19239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:09.304096 19239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:09.304225 19239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:23:09.306173 19239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:09.306553 19239 leader_election.cc:302] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca098de226bc43b3beabb054230000ed; no voters:
I20260929 16:23:09.306902 19239 leader_election.cc:288] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:23:09.306988 19243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:23:09.307188 19243 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 LEADER]: Becoming Leader. State: Replica: ca098de226bc43b3beabb054230000ed, State: Running, Role: LEADER
I20260929 16:23:09.307487 19243 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:09.308152 19239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:09.308439 19243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca098de226bc43b3beabb054230000ed. Latest consensus state: current_term: 2 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:09.308558 19243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:09.308898 19247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:09.308983 19244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:09.309170 19244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:09.313721 19247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:09.314458 19247 catalog_manager.cc:1259] Loaded cluster ID: 750eab6f9e944aa8aab91499a92d26a2
I20260929 16:23:09.314515 19247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:09.315903 19247 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:23:09.320667 19247 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:23:09.326646 19247 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:09.327521 19247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Loaded TSK: 0
I20260929 16:23:09.328787 19247 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:23:09.387665 19241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:09.387969 19241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:09.388053 19241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:09.392200 19241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:09.392290 19241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:09.392386 19241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:23:09.397917 19241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:44275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=44547
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:09.399094 19241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:09.400149 19241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:09.410840 19266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.410846 19267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.411329 19269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:09.411410 19241 server_base.cc:1034] running on GCE node
I20260929 16:23:09.411885 19241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:09.412595 19241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:09.413800 19241 hybrid_clock.cc:648] HybridClock initialized: now 1790698989413763 us; error 53 us; skew 500 ppm
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:09.416868 19241 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:23:09.417860 19241 webserver.cc:460] Webserver started at http://127.4.200.65:44547/ using document root <none> and password file <none>
I20260929 16:23:09.418498 19241 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:09.418607 19241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:09.422897 19241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:23:09.425194 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.426563 19241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:23:09.426694 19241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "826b6a4a29c641aca711b18274172d01"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "a6ea001f4711c9e8fc52a0a6add07542"
server_key_iv: "4a04f32a9119830dcd90564ae5502d3c"
server_key_version: "encryptionkey@0"
I20260929 16:23:09.427238 19241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:09.444001 19241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:09.447324 19241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:09.447629 19241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:09.448287 19241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:09.449429 19283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:23:09.455147 19241 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:23:09.455259 19241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:23:09.455349 19241 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:23:09.457338 19241 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:23:09.457407 19241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:23:09.457851 19283 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Bootstrap starting.
I20260929 16:23:09.468631 19241 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44275
I20260929 16:23:09.468643 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44275 every 8 connection(s)
I20260929 16:23:09.469898 19241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:23:09.477366 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19241
I20260929 16:23:09.477885 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:46037
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=44199
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:09.478434 19283 log.cc:824] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:09.487263 19283 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:09.487711 19283 tablet_bootstrap.cc:492] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Bootstrap complete.
I20260929 16:23:09.488492 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.472109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:49407 (local address 127.4.200.126:39475)
0929 16:23:09.472485 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.472511 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.472859 (+ 348us) server_negotiation.cc:407] Connection header received
0929 16:23:09.473983 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.473995 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.474289 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.474677 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:09.475806 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.476790 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.478614 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.479224 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.480066 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.482250 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:09.482257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:09.482274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:09.482308 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:09.484442 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.485186 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.485192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.485198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.485347 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.485786 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.485789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.485791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.486230 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:09.486364 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.487120 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.487539 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":131,"threads_started":1}
I20260929 16:23:09.488543 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.472112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:49407)
0929 16:23:09.472684 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.472726 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.473558 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.474731 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.474739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.475161 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:09.475627 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.475642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.476971 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.476974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.478418 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.478434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.479450 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.479456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.479684 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.480393 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:09.480413 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:09.482043 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:09.484587 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.484598 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.484615 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.485050 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.485467 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.485471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.485474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.485674 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.486364 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:09.486374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:09.486737 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.486982 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.487377 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
I20260929 16:23:09.489168 19283 ts_tablet_manager.cc:1397] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260929 16:23:09.491938 19391 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:09.492182 19391 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:09.491873 19283 raft_consensus.cc:348] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:09.492439 19283 raft_consensus.cc:729] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826b6a4a29c641aca711b18274172d01, State: Initialized, Role: FOLLOWER
I20260929 16:23:09.492726 19391 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:09.492936 19283 consensus_queue.cc:260] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } } peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } }
I20260929 16:23:09.493949 19283 ts_tablet_manager.cc:1428] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 16:23:09.495688 19203 ts_manager.cc:194] Registered new tserver with Master: 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275)
I20260929 16:23:09.499063 19203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:49407
I20260929 16:23:09.501777 19391 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:23:09.509744 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.498527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44275 (local address 127.0.0.1:54202)
0929 16:23:09.499237 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.499280 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.499418 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.499938 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.499943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.500396 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:09.500535 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.500543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.502002 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.502005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.502959 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.502973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.503715 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.503717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.503797 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.504401 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:09.504413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:09.505932 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:09.508054 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.508062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.508068 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.508320 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.508630 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.508633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.508634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.508712 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.509207 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:09.509212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:09.509310 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.509443 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.509598 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":100,"threads_started":1}
I20260929 16:23:09.509881 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.498602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.4.200.65:44275)
0929 16:23:09.498953 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.498966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.499388 (+ 422us) server_negotiation.cc:407] Connection header received
0929 16:23:09.499570 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.499577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.499783 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.499923 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:09.500677 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.501869 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.503104 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.503587 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.503695 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.506073 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:09.506084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:09.506096 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:09.506123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:09.507924 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.508440 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.508446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.508448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.508541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.508796 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.508799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.508801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.509091 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:09.509221 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.509533 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.509704 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":93,"threads_started":1}
I20260929 16:23:09.510888 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c26c3ff9ae34ee1b369703c319ab650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:09 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:54202
I20260929 16:23:09.511806 19400 tablet_replica.cc:330] stopping tablet replica
I20260929 16:23:09.512163 19400 raft_consensus.cc:2229] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:09.512472 19400 raft_consensus.cc:2258] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:09.513417 19400 ts_tablet_manager.cc:1905] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:09.516989 19400 ts_tablet_manager.cc:1918] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:09.517107 19400 log.cc:1196] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/0c26c3ff9ae34ee1b369703c319ab650
I20260929 16:23:09.517385 19400 ts_tablet_manager.cc:1939] T 0c26c3ff9ae34ee1b369703c319ab650 P 826b6a4a29c641aca711b18274172d01: Deleting consensus metadata
I20260929 16:23:09.519044 19189 catalog_manager.cc:4841] TS 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275): tablet 0c26c3ff9ae34ee1b369703c319ab650 (table test-table [id=ec9faf0326fb4f0cb9dbc5e1be205565]) successfully deleted
W20260929 16:23:09.600088 19396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:09.600381 19396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:09.600427 19396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:09.604585 19396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:09.604647 19396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:09.604755 19396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:23:09.610477 19396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:46037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=44199
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:09.611626 19396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:09.612720 19396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:09.622879 19406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.622882 19408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.622882 19405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:09.623272 19396 server_base.cc:1034] running on GCE node
I20260929 16:23:09.623730 19396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:09.624380 19396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:09.625587 19396 hybrid_clock.cc:648] HybridClock initialized: now 1790698989625543 us; error 63 us; skew 500 ppm
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:09.628289 19396 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:23:09.629208 19396 webserver.cc:460] Webserver started at http://127.4.200.66:44199/ using document root <none> and password file <none>
I20260929 16:23:09.629776 19396 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:09.629899 19396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:09.633872 19396 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:09.636245 19415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.637490 19396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:23:09.637629 19396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "b49f71eee20b41399f9cf8ada7807abc"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "709e731ceb1341f7cc338cb3fa21a9c4"
server_key_iv: "3a55f19c5c1f6593edfda3a2f2823a1c"
server_key_version: "encryptionkey@0"
I20260929 16:23:09.638188 19396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:09.655010 19396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:09.658157 19396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:09.658370 19396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:09.659014 19396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:09.659895 19396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:09.659942 19396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.660003 19396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:09.660043 19396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.668820 19396 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:46037
I20260929 16:23:09.668874 19528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:46037 every 8 connection(s)
I20260929 16:23:09.669862 19396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:23:09.676937 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19396
I20260929 16:23:09.677500 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:36111
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=35429
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:09.685560 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.671789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:55437 (local address 127.4.200.126:39475)
0929 16:23:09.671964 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.671970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.672857 (+ 887us) server_negotiation.cc:407] Connection header received
0929 16:23:09.674555 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.674558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.674672 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.674779 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:09.676272 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.676911 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.678718 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.678968 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.679104 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.681670 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:09.681673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:09.681675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:09.681700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:09.683390 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.683961 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.683965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.683973 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.684058 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.684412 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.684417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.684418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.684617 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:09.684729 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.685249 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.685389 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:23:09.686489 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.672211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:55437)
0929 16:23:09.672700 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.672742 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.674270 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.675034 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.675044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.675532 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:09.676081 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.676102 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.677052 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.677057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.678569 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.678595 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.679144 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.679149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.679345 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.680002 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:09.680018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:09.681478 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:09.683548 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.683554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.683571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.683845 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.684162 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.684165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.684167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.684314 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.684728 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:09.684734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:09.685002 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.685210 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.685547 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
I20260929 16:23:09.687850 19529 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:09.688177 19529 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:09.688936 19529 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:09.690287 19203 ts_manager.cc:194] Registered new tserver with Master: b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037)
I20260929 16:23:09.691171 19203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:55437
W20260929 16:23:09.812945 19532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:09.813215 19532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:09.813259 19532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:09.817344 19532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:09.817411 19532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:09.817471 19532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:23:09.822917 19532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:36111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=35429
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:09.823961 19532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:09.825080 19532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:09.834883 19538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.834889 19540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:09.834936 19537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:09.835142 19532 server_base.cc:1034] running on GCE node
I20260929 16:23:09.835539 19532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:09.836251 19532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:09.837442 19532 hybrid_clock.cc:648] HybridClock initialized: now 1790698989837425 us; error 30 us; skew 500 ppm
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:09.840160 19532 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:23:09.841053 19532 webserver.cc:460] Webserver started at http://127.4.200.67:35429/ using document root <none> and password file <none>
I20260929 16:23:09.841677 19532 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:09.841786 19532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:09.845916 19532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:23:09.848340 19547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.849622 19532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:09.849761 19532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "c0304336845a561559f9a780536bc227"
server_key_iv: "4cbd663ef3ce29153f2dc3cbd18bf071"
server_key_version: "encryptionkey@0"
I20260929 16:23:09.850338 19532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:09.866868 19532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:09.869879 19532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:09.870091 19532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:09.870785 19532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:09.871865 19554 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 16:23:09.875680 19554 ts_tablet_manager.cc:1852] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 16:23:09.876058 19554 ts_tablet_manager.cc:1905] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:09.879757 19554 ts_tablet_manager.cc:1918] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:09.879848 19554 log.cc:1196] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/0c26c3ff9ae34ee1b369703c319ab650
I20260929 16:23:09.880226 19554 ts_tablet_manager.cc:1939] T 0c26c3ff9ae34ee1b369703c319ab650 P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting consensus metadata
I20260929 16:23:09.880530 19532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 16:23:09.880602 19532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260929 16:23:09.880702 19532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:09.880756 19532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:09.889724 19532 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:36111
I20260929 16:23:09.889760 19661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:36111 every 8 connection(s)
I20260929 16:23:09.890594 19532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:23:09.895236 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19532
Sep 29 16:23:09 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:09.904254 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.895889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.4.200.126:39475)
0929 16:23:09.896118 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.896122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.896255 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:23:09.896355 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.896357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.896408 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.896498 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:09.897385 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.897902 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.899006 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.899277 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.899360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.900690 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:09.900694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:09.900697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:09.900722 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:09.902613 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.903024 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.903029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.903031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.903108 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.903396 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.903400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.903402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.903577 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:09.903782 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.903931 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.904047 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260929 16:23:09.904632 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.892307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:37763 (local address 127.4.200.126:39475)
0929 16:23:09.892472 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.892477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.893331 (+ 854us) server_negotiation.cc:407] Connection header received
0929 16:23:09.894213 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.894216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.894261 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.894339 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:09.895635 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.896376 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.897861 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.898140 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.898213 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.901118 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:09.901121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:09.901123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:09.901154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:09.902671 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.903184 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.903189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.903191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.903268 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:09.903616 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:09.903621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:09.903623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:09.903779 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:09.903880 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.904393 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.904498 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:23:09.905619 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.892697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:37763)
0929 16:23:09.893179 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.893221 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.894022 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.894567 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.894575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.894927 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:09.895483 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.895498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.896539 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.896543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.897746 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.897756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.898280 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.898305 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.898526 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.899178 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:09.899194 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:09.900944 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:09.902787 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.902793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.902808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.903077 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.903368 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:09.903372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:09.903374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:09.903519 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:09.903895 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:09.903903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:09.904146 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.904335 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.904672 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260929 16:23:09.906915 19662 heartbeater.cc:347] Connected to a master server at 127.4.200.126:39475
I20260929 16:23:09.907159 19662 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:09.907649 19662 heartbeater.cc:511] Master 127.4.200.126:39475 requested a full tablet report, sending...
I20260929 16:23:09.908722 19203 ts_manager.cc:194] Registered new tserver with Master: 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
I20260929 16:23:09.909350 19203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:37763
I20260929 16:23:09.918542 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:23:09.923115 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.919212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.4.200.126:39475)
0929 16:23:09.919350 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.919354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.919538 (+ 184us) server_negotiation.cc:407] Connection header received
0929 16:23:09.919691 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.919693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.919799 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.919899 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:09.920160 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.920831 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.921682 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.921910 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.921988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.922307 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:09.922783 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:09.922887 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.922954 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.923005 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:23:09.923803 19203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:23:09.927234 19203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:23:09.935103 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 60cb841df5ad4aa7ac972ff3450c734f (DEFAULT_TABLE table=test-table [id=010defdf0f1f4bfe86670e3f8be89176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:09.935811 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cb841df5ad4aa7ac972ff3450c734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:09.938472 19283 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Bootstrap starting.
I20260929 16:23:09.939414 19283 tablet_bootstrap.cc:654] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:09.940423 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.934303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:36111 (local address 127.0.0.1:60000)
0929 16:23:09.934661 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.934676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.934761 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.935542 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.935545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.935572 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:09.935719 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.935726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.937266 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.937269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.938830 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.938841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.939952 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.939956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.940043 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.940086 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.940171 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.940252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":217,"spinlock_wait_cycles":1152,"thread_start_us":134,"threads_started":1}
I20260929 16:23:09.940440 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.934570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.4.200.67:36111)
0929 16:23:09.935018 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.935030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.935055 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:23:09.935131 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.935139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.935349 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.935867 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:09.936019 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.937088 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.938979 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.939821 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.939943 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.940095 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.940202 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.940289 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":163,"threads_started":1}
I20260929 16:23:09.940785 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.934029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46037 (local address 127.0.0.1:58990)
0929 16:23:09.934155 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:09.934172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:09.934257 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:09.935536 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:09.935539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:09.935569 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:09.935701 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.935711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.937213 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.937219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.939035 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:09.939043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.940321 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.940325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:09.940449 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.940482 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:23:09.940542 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 16:23:09.940638 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":334,"spinlock_wait_cycles":4864}
I20260929 16:23:09.941425 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:09.934604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.4.200.66:46037)
0929 16:23:09.935034 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:09.935042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:09.935062 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:23:09.935125 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:09.935130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:09.935338 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:09.935565 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:09.935803 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.937086 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.939136 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:09.940175 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:09.940959 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:09.941099 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:09.941149 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:23:09.941236 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":125,"threads_started":1}
I20260929 16:23:09.941918 19283 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: No bootstrap required, opened a new log
I20260929 16:23:09.942008 19283 ts_tablet_manager.cc:1397] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:23:09.942505 19283 raft_consensus.cc:348] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.942633 19283 raft_consensus.cc:374] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:09.942075 19596 tablet_service.cc:1467] Processing CreateTablet for tablet 60cb841df5ad4aa7ac972ff3450c734f (DEFAULT_TABLE table=test-table [id=010defdf0f1f4bfe86670e3f8be89176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:09.942674 19283 raft_consensus.cc:729] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826b6a4a29c641aca711b18274172d01, State: Initialized, Role: FOLLOWER
I20260929 16:23:09.942799 19283 consensus_queue.cc:260] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.943050 19283 ts_tablet_manager.cc:1428] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:23:09.943302 19596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cb841df5ad4aa7ac972ff3450c734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:09.944489 19463 tablet_service.cc:1467] Processing CreateTablet for tablet 60cb841df5ad4aa7ac972ff3450c734f (DEFAULT_TABLE table=test-table [id=010defdf0f1f4bfe86670e3f8be89176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:09.945444 19463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cb841df5ad4aa7ac972ff3450c734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:09.948022 19554 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Bootstrap starting.
I20260929 16:23:09.949901 19554 tablet_bootstrap.cc:654] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:09.950695 19554 log.cc:824] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:09.952508 19554 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: No bootstrap required, opened a new log
I20260929 16:23:09.952507 19676 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Bootstrap starting.
I20260929 16:23:09.952725 19554 ts_tablet_manager.cc:1397] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:23:09.954960 19676 tablet_bootstrap.cc:654] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:09.955814 19676 log.cc:824] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:09.956202 19554 raft_consensus.cc:348] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.956493 19554 raft_consensus.cc:374] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:09.956560 19554 raft_consensus.cc:729] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec09ed46c5a4dc4922db6086a21de6b, State: Initialized, Role: FOLLOWER
I20260929 16:23:09.957015 19554 consensus_queue.cc:260] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.957659 19662 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
I20260929 16:23:09.957837 19676 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: No bootstrap required, opened a new log
I20260929 16:23:09.957890 19554 ts_tablet_manager.cc:1428] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:23:09.958011 19676 ts_tablet_manager.cc:1397] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:23:09.961728 19676 raft_consensus.cc:348] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.961961 19676 raft_consensus.cc:374] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:09.961995 19676 raft_consensus.cc:729] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49f71eee20b41399f9cf8ada7807abc, State: Initialized, Role: FOLLOWER
I20260929 16:23:09.962369 19676 consensus_queue.cc:260] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:09.963050 19529 heartbeater.cc:503] Master 127.4.200.126:39475 was elected leader, sending a full tablet report...
I20260929 16:23:09.963282 19676 ts_tablet_manager.cc:1428] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 16:23:09.971197 19392 tablet.cc:2369] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:10.132999 19397 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:10.133229 19397 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:10.134344 19397 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
I20260929 16:23:10.139780 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.135208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40879 (local address 127.4.200.67:36111)
0929 16:23:10.135356 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:10.135360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:10.135439 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:23:10.135530 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:10.135533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:10.135573 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:10.135637 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:10.135899 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.136621 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.138324 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.138967 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.139094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:10.139151 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:10.139573 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 16:23:10.139648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:23:10.139780 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.134764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:41837 (local address 127.4.200.66:46037)
0929 16:23:10.134913 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:10.134916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:10.135143 (+ 227us) server_negotiation.cc:407] Connection header received
0929 16:23:10.135210 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:10.135213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:10.135271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:10.135369 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:10.135732 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.136557 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.138320 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.138967 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.139094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:10.139151 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:10.139572 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 16:23:10.139648 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:23:10.139904 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.134625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46037 (local address 127.4.200.65:41837)
0929 16:23:10.134927 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.134943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:10.135069 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:10.135399 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:10.135402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:10.135426 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:10.135583 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:10.135588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.136702 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.136707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:10.138184 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:10.138195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.139127 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.139130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:10.139323 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:10.139445 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 16:23:10.139545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:23:10.139705 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":244,"thread_start_us":122,"threads_started":1}
I20260929 16:23:10.139904 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.135050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:36111 (local address 127.4.200.65:40879)
0929 16:23:10.135325 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.135362 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:10.135442 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:10.135677 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:10.135680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:10.135701 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:10.135782 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:10.135787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.136715 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.136718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:10.138191 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:10.138197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.139125 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.139130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:10.139306 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:10.139445 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 16:23:10.139545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:23:10.139705 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":180,"thread_start_us":66,"threads_started":1}
I20260929 16:23:10.140738 19616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cb841df5ad4aa7ac972ff3450c734f" candidate_uuid: "826b6a4a29c641aca711b18274172d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" is_pre_election: true
I20260929 16:23:10.140738 19483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cb841df5ad4aa7ac972ff3450c734f" candidate_uuid: "826b6a4a29c641aca711b18274172d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49f71eee20b41399f9cf8ada7807abc" is_pre_election: true
I20260929 16:23:10.141078 19483 raft_consensus.cc:2454] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826b6a4a29c641aca711b18274172d01 in term 0.
I20260929 16:23:10.141078 19616 raft_consensus.cc:2454] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 826b6a4a29c641aca711b18274172d01 in term 0.
W20260929 16:23:10.141467 19663 tablet.cc:2369] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:10.141611 19280 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826b6a4a29c641aca711b18274172d01, b49f71eee20b41399f9cf8ada7807abc; no voters:
I20260929 16:23:10.141916 19397 raft_consensus.cc:2781] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:10.142004 19397 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:10.142053 19397 raft_consensus.cc:3037] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:10.143319 19397 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:10.143721 19397 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 1 election: Requested vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
I20260929 16:23:10.144115 19483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cb841df5ad4aa7ac972ff3450c734f" candidate_uuid: "826b6a4a29c641aca711b18274172d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49f71eee20b41399f9cf8ada7807abc"
I20260929 16:23:10.144135 19616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cb841df5ad4aa7ac972ff3450c734f" candidate_uuid: "826b6a4a29c641aca711b18274172d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
I20260929 16:23:10.144251 19483 raft_consensus.cc:3037] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:10.144250 19616 raft_consensus.cc:3037] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:10.145536 19616 raft_consensus.cc:2454] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826b6a4a29c641aca711b18274172d01 in term 1.
I20260929 16:23:10.145536 19483 raft_consensus.cc:2454] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826b6a4a29c641aca711b18274172d01 in term 1.
I20260929 16:23:10.145836 19280 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826b6a4a29c641aca711b18274172d01, b49f71eee20b41399f9cf8ada7807abc; no voters:
I20260929 16:23:10.146039 19397 raft_consensus.cc:2781] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:10.146234 19397 raft_consensus.cc:686] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 LEADER]: Becoming Leader. State: Replica: 826b6a4a29c641aca711b18274172d01, State: Running, Role: LEADER
I20260929 16:23:10.146552 19397 consensus_queue.cc:237] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:10.149652 19203 catalog_manager.cc:5495] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 826b6a4a29c641aca711b18274172d01 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "826b6a4a29c641aca711b18274172d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } health_report { overall_health: HEALTHY } } }
W20260929 16:23:10.171168 19530 tablet.cc:2369] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:10.181180 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.177500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.4.200.65:44275)
0929 16:23:10.177788 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:10.177795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:10.177808 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:23:10.177881 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:10.177884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:10.177937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:10.178038 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:10.178288 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.178852 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.179796 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:10.180026 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:10.180107 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:10.180346 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:10.180781 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:10.180867 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:10.180924 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:23:10.180971 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
I20260929 16:23:10.185344 19616 raft_consensus.cc:1261] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Refusing update from remote peer 826b6a4a29c641aca711b18274172d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:10.185345 19483 raft_consensus.cc:1261] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Refusing update from remote peer 826b6a4a29c641aca711b18274172d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:10.186116 19682 consensus_queue.cc:1035] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:10.186316 19397 consensus_queue.cc:1035] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:10.193930 19686 mvcc.cc:205] Tried to move back new op lower bound from 7334703063791788032 to 7334703063643058176. Current Snapshot: MvccSnapshot[applied={T|T < 7334703063791788032}]
I20260929 16:23:10.194291 19688 mvcc.cc:205] Tried to move back new op lower bound from 7334703063791788032 to 7334703063643058176. Current Snapshot: MvccSnapshot[applied={T|T < 7334703063791788032}]
I20260929 16:23:10.196800 19687 mvcc.cc:205] Tried to move back new op lower bound from 7334703063791788032 to 7334703063643058176. Current Snapshot: MvccSnapshot[applied={T|T < 7334703063791788032}]
I20260929 16:23:10.202512 19202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:10.202674 19202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:10.205089 19202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Sending DeleteTablet for 3 replicas of tablet 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:10.205827 19463 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cb841df5ad4aa7ac972ff3450c734f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:10 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:58990
I20260929 16:23:10.205932 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19241
I20260929 16:23:10.205973 19596 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cb841df5ad4aa7ac972ff3450c734f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:10 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260929 16:23:10.207077 19696 tablet_replica.cc:330] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: stopping tablet replica
I20260929 16:23:10.207455 19696 raft_consensus.cc:2229] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:10.207718 19695 tablet_replica.cc:330] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: stopping tablet replica
I20260929 16:23:10.207803 19696 raft_consensus.cc:2258] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:10.208019 19695 raft_consensus.cc:2229] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:10.208353 19695 raft_consensus.cc:2258] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:10.209543 19696 ts_tablet_manager.cc:1905] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:10.209761 19695 ts_tablet_manager.cc:1905] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:10.212133 19696 ts_tablet_manager.cc:1918] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:10.212225 19696 log.cc:1196] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:10.212481 19696 ts_tablet_manager.cc:1939] T 60cb841df5ad4aa7ac972ff3450c734f P 6ec09ed46c5a4dc4922db6086a21de6b: Deleting consensus metadata
I20260929 16:23:10.212790 19695 ts_tablet_manager.cc:1918] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:10.212891 19695 log.cc:1196] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:10.213195 19695 ts_tablet_manager.cc:1939] T 60cb841df5ad4aa7ac972ff3450c734f P b49f71eee20b41399f9cf8ada7807abc: Deleting consensus metadata
W20260929 16:23:10.213621 19189 connection.cc:535] client connection to 127.4.200.65:44275 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:23:10.213693 19190 catalog_manager.cc:4841] TS 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): tablet 60cb841df5ad4aa7ac972ff3450c734f (table test-table [id=010defdf0f1f4bfe86670e3f8be89176]) successfully deleted
W20260929 16:23:10.213769 19189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:23:10.214193 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19396
I20260929 16:23:10.214823 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.214482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:44275 (local address 127.0.0.1:54222)
0929 16:23:10.214658 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.214729 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:23:10.214959 19189 catalog_manager.cc:4584] TS 826b6a4a29c641aca711b18274172d01 (127.4.200.65:44275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60cb841df5ad4aa7ac972ff3450c734f: Network error: Client connection negotiation failed: client connection to 127.4.200.65:44275: connect: Connection refused (error 111)
I20260929 16:23:10.217007 19190 catalog_manager.cc:4841] TS b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): tablet 60cb841df5ad4aa7ac972ff3450c734f (table test-table [id=010defdf0f1f4bfe86670e3f8be89176]) successfully deleted
I20260929 16:23:10.221232 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19532
I20260929 16:23:10.226806 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19170
I20260929 16:23:10.233459 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--webserver_interface=127.4.200.126
--webserver_port=40999
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:10.343012 19698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:10.343320 19698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:10.343394 19698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:10.347474 19698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:10.347553 19698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:10.347589 19698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:10.347651 19698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:10.347707 19698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:10.353012 19698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=40999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:10.354058 19698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:10.355156 19698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:10.363554 19706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:10.363554 19703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.363643 19698 server_base.cc:1034] running on GCE node
W20260929 16:23:10.363801 19704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.364393 19698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:10.365506 19698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:10.366664 19698 hybrid_clock.cc:648] HybridClock initialized: now 1790698990366649 us; error 26 us; skew 500 ppm
Sep 29 16:23:10 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:10.369424 19698 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:10.370296 19698 webserver.cc:460] Webserver started at http://127.4.200.126:40999/ using document root <none> and password file <none>
I20260929 16:23:10.370873 19698 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:10.370936 19698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:10.374724 19698 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:23:10.376762 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.377938 19698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:10.378062 19698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "ca098de226bc43b3beabb054230000ed"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "8646cdeeb2b490f6871248e5098c63de"
server_key_iv: "a8803703cfc8ec179c6ab1c1815f29ea"
server_key_version: "encryptionkey@0"
I20260929 16:23:10.378654 19698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:10.396932 19698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:10.399796 19698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:10.399999 19698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:10.407070 19698 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:39475
I20260929 16:23:10.407347 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:39475 every 8 connection(s)
I20260929 16:23:10.408210 19698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:10.409926 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19698
I20260929 16:23:10.410415 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:44275
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=44547
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:10.413187 19766 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:23:10.414899 19766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap starting.
I20260929 16:23:10.432279 19766 log.cc:824] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:10.443439 19766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:10.443871 19766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: Bootstrap complete.
I20260929 16:23:10.448083 19766 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:10.448477 19766 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca098de226bc43b3beabb054230000ed, State: Initialized, Role: FOLLOWER
I20260929 16:23:10.448966 19766 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:10.449081 19766 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:10.449167 19766 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:10.449287 19766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:23:10.451258 19766 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:10.451588 19766 leader_election.cc:302] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca098de226bc43b3beabb054230000ed; no voters:
I20260929 16:23:10.451944 19766 leader_election.cc:288] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:23:10.452008 19770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:23:10.452267 19770 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [term 3 LEADER]: Becoming Leader. State: Replica: ca098de226bc43b3beabb054230000ed, State: Running, Role: LEADER
I20260929 16:23:10.452558 19770 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } }
I20260929 16:23:10.453255 19766 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:10.453539 19772 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:10.453656 19772 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:10.454002 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca098de226bc43b3beabb054230000ed. Latest consensus state: current_term: 3 leader_uuid: "ca098de226bc43b3beabb054230000ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca098de226bc43b3beabb054230000ed" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39475 } } }
I20260929 16:23:10.458891 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:10.464129 19785 catalog_manager.cc:1259] Loaded cluster ID: 750eab6f9e944aa8aab91499a92d26a2
I20260929 16:23:10.464214 19785 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: loading cluster ID for follower catalog manager: success
W20260929 16:23:10.465337 19785 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ca098de226bc43b3beabb054230000ed: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260929 16:23:10.465415 19785 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260929 16:23:10.465468 19774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:10.468297 19774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:10.468807 19774 catalog_manager.cc:1259] Loaded cluster ID: 750eab6f9e944aa8aab91499a92d26a2
I20260929 16:23:10.468964 19774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:23:10.469650 19774 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f372ca539ad google::LogMessage::Fail() at ??:0
@ 0x7f372ca5688a google::LogMessage::SendToLog() at ??:0
@ 0x7f372ca534de google::LogMessage::Flush() at ??:0
@ 0x7f372ca54a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3737392fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3737393d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3737390579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f37373d4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f372eca53d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f372eca5c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f372eca73f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f372ec96512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f37302da6db start_thread at ??:0
@ 0x7f372b55161f clone at ??:0
W20260929 16:23:10.552094 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:10.552364 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:10.552409 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:10.556576 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:10.556640 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:10.556701 19768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:23:10.562278 19768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:44275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=44547
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:10.563495 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:10.564529 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:10.575189 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.575253 19768 server_base.cc:1034] running on GCE node
W20260929 16:23:10.575186 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:10.575495 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.575997 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:10.576633 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:10.577808 19768 hybrid_clock.cc:648] HybridClock initialized: now 1790698990577785 us; error 44 us; skew 500 ppm
Sep 29 16:23:10 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:10.580612 19768 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:23:10.581439 19768 webserver.cc:460] Webserver started at http://127.4.200.65:44547/ using document root <none> and password file <none>
I20260929 16:23:10.581995 19768 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:10.582098 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:10.585991 19768 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:23:10.588253 19817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.589515 19768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:23:10.589648 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "826b6a4a29c641aca711b18274172d01"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "a6ea001f4711c9e8fc52a0a6add07542"
server_key_iv: "4a04f32a9119830dcd90564ae5502d3c"
server_key_version: "encryptionkey@0"
I20260929 16:23:10.590185 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:10.598979 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:10.602020 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:10.602216 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:10.602838 19768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:10.603910 19824 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:23:10.609201 19768 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:23:10.609282 19768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:23:10.609391 19768 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:23:10.611032 19768 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:23:10.611102 19768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 16:23:10.611456 19824 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Bootstrap starting.
I20260929 16:23:10.622212 19768 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:44275
I20260929 16:23:10.622483 19931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:44275 every 8 connection(s)
I20260929 16:23:10.623268 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:23:10.632989 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19768
I20260929 16:23:10.633206 19824 log.cc:824] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:10.633489 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:46037
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=44199
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:10.642588 19824 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:10.643043 19824 tablet_bootstrap.cc:492] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Bootstrap complete.
I20260929 16:23:10.644495 19824 ts_tablet_manager.cc:1397] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.009s
I20260929 16:23:10.647817 19824 raft_consensus.cc:348] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:10.648362 19824 raft_consensus.cc:729] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826b6a4a29c641aca711b18274172d01, State: Initialized, Role: FOLLOWER
I20260929 16:23:10.648908 19824 consensus_queue.cc:260] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:10.649981 19824 ts_tablet_manager.cc:1428] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260929 16:23:10.685882 19934 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:23:10.625193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:47437)
0929 16:23:10.625680 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.625732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:10.626618 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:10.684913 (+ 58295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":158,"threads_started":1}
W20260929 16:23:10.686525 19932 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:39475 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:39475: Client connection negotiation failed: client connection to 127.4.200.126:39475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:23:10.687270 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.686989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:36959)
0929 16:23:10.687127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.687204 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:23:10.687933 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.687715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:42815)
0929 16:23:10.687830 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.687878 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:23:10.688097 19932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:23:10.751130 19936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:10.751394 19936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:10.751437 19936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:10.755393 19936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:10.755460 19936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:10.755517 19936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:23:10.760681 19936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:46037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=44199
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:10.761718 19936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:10.762830 19936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:10.772706 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.772971 19936 server_base.cc:1034] running on GCE node
W20260929 16:23:10.772706 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:10.772719 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.773520 19936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:10.774153 19936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:10.775308 19936 hybrid_clock.cc:648] HybridClock initialized: now 1790698990775278 us; error 40 us; skew 500 ppm
Sep 29 16:23:10 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:10.777940 19936 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:23:10.778916 19936 webserver.cc:460] Webserver started at http://127.4.200.66:44199/ using document root <none> and password file <none>
I20260929 16:23:10.779488 19936 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:10.779549 19936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:10.783437 19936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:23:10.785528 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.786820 19936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:10.786966 19936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "b49f71eee20b41399f9cf8ada7807abc"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "709e731ceb1341f7cc338cb3fa21a9c4"
server_key_iv: "3a55f19c5c1f6593edfda3a2f2823a1c"
server_key_version: "encryptionkey@0"
I20260929 16:23:10.787532 19936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:10.803443 19936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:10.806602 19936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:10.806805 19936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:10.807411 19936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:10.808250 19936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:10.808331 19936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.808423 19936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:10.808475 19936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.818168 19936 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:46037
I20260929 16:23:10.818192 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:46037 every 8 connection(s)
I20260929 16:23:10.819269 19936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:23:10.823009 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 19936
I20260929 16:23:10.823432 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:36111
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:39475
--webserver_port=35429
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:10.823630 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.821436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:54097)
0929 16:23:10.822008 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.822136 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":168,"threads_started":1}
W20260929 16:23:10.824177 20066 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:39475 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:39475: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
I20260929 16:23:10.824894 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.824632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:41561)
0929 16:23:10.824773 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.824826 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:23:10.825560 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:10.825357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:56073)
0929 16:23:10.825461 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:10.825516 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:23:10.825798 20066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:23:10.936573 20069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:10.936875 20069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:10.936921 20069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:10.941131 20069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:10.941198 20069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:10.941260 20069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:23:10.946834 20069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:36111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=35429
--tserver_master_addrs=127.4.200.126:39475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:10.947963 20069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:10.949026 20069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:10.959723 20077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:10.959702 20074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.959856 20069 server_base.cc:1034] running on GCE node
W20260929 16:23:10.959702 20075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:10.960527 20069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:10.961174 20069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:10.962344 20069 hybrid_clock.cc:648] HybridClock initialized: now 1790698990962324 us; error 34 us; skew 500 ppm
Sep 29 16:23:10 dist-test-slave-s9ct krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:10.965212 20069 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:23:10.966111 20069 webserver.cc:460] Webserver started at http://127.4.200.67:35429/ using document root <none> and password file <none>
I20260929 16:23:10.966761 20069 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:10.966858 20069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:10.970835 20069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:23:10.972968 20084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.974218 20069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:10.974370 20069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "6ec09ed46c5a4dc4922db6086a21de6b"
format_stamp: "Formatted at 2026-09-29 16:23:08 on dist-test-slave-s9ct"
server_key: "c0304336845a561559f9a780536bc227"
server_key_iv: "4cbd663ef3ce29153f2dc3cbd18bf071"
server_key_version: "encryptionkey@0"
I20260929 16:23:10.974995 20069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:10.988426 20069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:10.991941 20069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:10.992153 20069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:10.992786 20069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:10.993746 20069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:10.993815 20069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:10.993882 20069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:10.993927 20069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:11.003050 20069 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:36111
I20260929 16:23:11.003072 20197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:36111 every 8 connection(s)
I20260929 16:23:11.004115 20069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:23:11.008247 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:11.006497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:44015)
0929 16:23:11.006968 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:11.007079 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
W20260929 16:23:11.008726 20198 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:39475 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:39475: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
I20260929 16:23:11.009327 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:11.009122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:36863)
0929 16:23:11.009225 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:11.009272 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:23:11.009915 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:11.009734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:51705)
0929 16:23:11.009831 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:11.009871 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 16:23:11.010105 20198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:23:11.010715 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20069
I20260929 16:23:11.689136 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:11.688709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:37643)
0929 16:23:11.688962 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:11.689021 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260929 16:23:11.826974 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:11.826493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:54067)
0929 16:23:11.826797 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:11.826865 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
I20260929 16:23:12.011197 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.010729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:35663)
0929 16:23:12.010967 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:12.011026 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260929 16:23:12.234606 20205 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:12.234794 20205 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:12.235925 20205 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
Sep 29 16:23:12 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.67@KRBTEST.COM
Sep 29 16:23:12 dist-test-slave-s9ct krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790698990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.66@KRBTEST.COM
I20260929 16:23:12.250559 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.236390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:56175 (local address 127.4.200.66:46037)
0929 16:23:12.236842 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:12.236867 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:12.236915 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:23:12.237592 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:12.237598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:12.237923 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:12.238204 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:12.239496 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.240449 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.241633 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.242069 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.242236 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:12.245276 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:12.245280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:12.245292 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:12.245319 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:12.247315 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248058 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:12.248065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:12.248071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:12.248233 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248672 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:12.248675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:12.248677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:12.249191 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:12.249336 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:12.250059 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 16:23:12.250361 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":122,"threads_started":1}
I20260929 16:23:12.250559 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.236687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:38467 (local address 127.4.200.67:36111)
0929 16:23:12.236989 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:12.237021 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:12.237053 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:23:12.237679 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:12.237684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:12.237964 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:12.238252 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:12.239497 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.240437 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.241629 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.242078 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.242255 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:12.245032 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:12.245039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:12.245055 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:12.245086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:12.247315 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248058 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:12.248065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:12.248071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:12.248233 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248672 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:12.248675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:12.248677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:12.249191 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:12.249329 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:12.250059 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 16:23:12.250350 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20260929 16:23:12.250545 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.236511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:36111 (local address 127.4.200.65:38467)
0929 16:23:12.236817 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:12.236831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:12.236919 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:12.238295 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:12.238302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:12.238845 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:12.239318 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:12.239333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.240585 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.240590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:12.241506 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:12.241515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.242223 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.242227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:12.242459 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:12.243160 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:12.243172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:12.244759 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:12.247449 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:12.247456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:12.247471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:12.247938 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:12.248337 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:12.248342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:12.248570 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:12.249313 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:12.249320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:12.249698 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 16:23:12.249911 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 16:23:12.250348 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":427,"spinlock_wait_cycles":41472,"thread_start_us":67,"threads_started":1}
I20260929 16:23:12.250545 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.236145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46037 (local address 127.4.200.65:56175)
0929 16:23:12.236446 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:12.236462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:12.236564 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:12.238297 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:12.238302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:12.238846 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:12.239319 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:12.239333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.240603 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.240606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:12.241507 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:12.241517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:12.242201 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:12.242204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:12.242423 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:12.243116 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:12.243134 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:12.245072 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:12.247449 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:12.247456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:12.247471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:12.247938 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:12.248337 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:12.248340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:12.248342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:12.248570 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:12.249321 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:12.249323 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:12.249698 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 16:23:12.249911 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 16:23:12.250297 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":191,"spinlock_wait_cycles":33280,"thread_start_us":71,"threads_started":1}
W20260929 16:23:12.253265 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:12.253451 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:12.253497 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:12.253721 20205 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:12.690245 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.689958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:55621)
0929 16:23:12.690100 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:12.690159 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:23:12.828330 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:12.827837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:42701)
0929 16:23:12.828166 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:12.828232 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
I20260929 16:23:13.012434 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:13.012033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:47475)
0929 16:23:13.012280 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:13.012334 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260929 16:23:13.691664 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:13.691175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:37789)
0929 16:23:13.691495 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:13.691561 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260929 16:23:13.829662 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:13.829244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:51533)
0929 16:23:13.829487 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:13.829547 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260929 16:23:14.013597 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:14.013167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:55771)
0929 16:23:14.013423 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:14.013485 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260929 16:23:14.085326 20215 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:14.085477 20215 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:14.085858 20215 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
W20260929 16:23:14.086799 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:14.086961 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:14.087018 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:14.087203 20215 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:14.693081 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:14.692578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:59039)
0929 16:23:14.692864 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:14.692922 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260929 16:23:14.830900 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:14.830493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:50173)
0929 16:23:14.830751 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:14.830805 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260929 16:23:15.014770 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:15.014343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:51759)
0929 16:23:15.014629 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:15.014680 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":141,"threads_started":1}
I20260929 16:23:15.590673 20219 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:15.590965 20219 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:15.591444 20219 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
W20260929 16:23:15.592154 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:15.592334 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:15.592378 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:15.592505 20219 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:15.694993 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:15.694542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:35809)
0929 16:23:15.694829 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:15.694895 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260929 16:23:15.832073 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:15.831633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:56127)
0929 16:23:15.831927 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:15.831979 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260929 16:23:16.015918 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:16.015517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:50403)
0929 16:23:16.015766 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:16.015827 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260929 16:23:16.696221 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:16.695783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:46077)
0929 16:23:16.696047 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:16.696108 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260929 16:23:16.833318 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:16.832907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:48295)
0929 16:23:16.833158 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:16.833221 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260929 16:23:17.016976 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:17.016605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:40315)
0929 16:23:17.016828 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:17.016882 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260929 16:23:17.697480 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:17.697038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:56543)
0929 16:23:17.697317 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:17.697377 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260929 16:23:17.834527 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:17.834019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:54389)
0929 16:23:17.834314 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:17.834374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
I20260929 16:23:18.018085 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:18.017664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:33033)
0929 16:23:18.017923 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:18.017991 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260929 16:23:18.522089 20229 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:18.522248 20229 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:18.522653 20229 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
W20260929 16:23:18.523461 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:18.523602 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:18.523661 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:18.523865 20229 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:18.698777 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:18.698285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:34269)
0929 16:23:18.698595 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:18.698678 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260929 16:23:18.835695 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:18.835244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:38399)
0929 16:23:18.835540 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:18.835599 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
I20260929 16:23:19.019479 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:19.018929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:52165)
0929 16:23:19.019306 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:19.019372 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":92,"threads_started":1}
I20260929 16:23:19.699826 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:19.699420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:43291)
0929 16:23:19.699674 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:19.699730 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260929 16:23:19.836957 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:19.836524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:50341)
0929 16:23:19.836792 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:19.836863 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260929 16:23:20.020819 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:20.020354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:57331)
0929 16:23:20.020664 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:20.020725 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":73,"threads_started":1}
I20260929 16:23:20.701151 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:20.700721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:34007)
0929 16:23:20.700997 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:20.701055 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260929 16:23:20.838135 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:20.837731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:60513)
0929 16:23:20.837982 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:20.838039 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260929 16:23:21.021963 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:21.021516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:55329)
0929 16:23:21.021803 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:21.021866 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":70,"threads_started":1}
I20260929 16:23:21.702301 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:21.701902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:52231)
0929 16:23:21.702148 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:21.702204 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260929 16:23:21.839361 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:21.838885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:60377)
0929 16:23:21.839166 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:21.839248 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260929 16:23:22.023142 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:22.022728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:51203)
0929 16:23:22.022991 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:22.023049 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260929 16:23:22.586275 20242 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:22.586441 20242 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:22.586869 20242 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
W20260929 16:23:22.587656 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:22.587772 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:22.587810 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:22.587939 20242 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:22.703464 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:22.703054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:43917)
0929 16:23:22.703310 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:22.703367 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260929 16:23:22.840521 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:22.840088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:48583)
0929 16:23:22.840352 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:22.840413 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260929 16:23:23.024384 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:23.023952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:54439)
0929 16:23:23.024218 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:23.024274 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260929 16:23:23.704826 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:23.704357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:58351)
0929 16:23:23.704656 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:23.704724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260929 16:23:23.841902 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:23.841349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:44037)
0929 16:23:23.841732 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:23.841801 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
I20260929 16:23:24.025683 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:24.025263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:45937)
0929 16:23:24.025526 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:24.025585 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260929 16:23:24.706061 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:24.705650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:50855)
0929 16:23:24.705902 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:24.705966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260929 16:23:24.843202 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:24.842740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:42727)
0929 16:23:24.843040 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:24.843103 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
I20260929 16:23:24.996423 20251 raft_consensus.cc:482] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:24.996554 20251 raft_consensus.cc:504] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b49f71eee20b41399f9cf8ada7807abc" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46037 } } peers { permanent_uuid: "6ec09ed46c5a4dc4922db6086a21de6b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 36111 } } peers { permanent_uuid: "826b6a4a29c641aca711b18274172d01" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 44275 } }
I20260929 16:23:24.997007 20251 leader_election.cc:288] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037), 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111)
W20260929 16:23:24.997833 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ec09ed46c5a4dc4922db6086a21de6b (127.4.200.67:36111): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
W20260929 16:23:24.997982 19821 leader_election.cc:341] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49f71eee20b41399f9cf8ada7807abc (127.4.200.66:46037): Not found: Tablet not found: 60cb841df5ad4aa7ac972ff3450c734f
I20260929 16:23:24.998054 19821 leader_election.cc:302] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 826b6a4a29c641aca711b18274172d01; no voters: 6ec09ed46c5a4dc4922db6086a21de6b, b49f71eee20b41399f9cf8ada7807abc
I20260929 16:23:24.998245 20251 raft_consensus.cc:2726] T 60cb841df5ad4aa7ac972ff3450c734f P 826b6a4a29c641aca711b18274172d01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:23:25.026919 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:25.026501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.67:40691)
0929 16:23:25.026768 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:25.026824 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20260929 16:23:25.707624 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:25.707115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.65:45169)
0929 16:23:25.707447 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:25.707526 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":91,"threads_started":1}
I20260929 16:23:25.844614 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:25.844197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39475 (local address 127.4.200.66:54147)
0929 16:23:25.844456 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:25.844515 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260929 16:23:26.013427 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19768
I20260929 16:23:26.019589 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 19936
I20260929 16:23:26.025132 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20069
2026-09-29T16:23:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19325 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:23:26 dist-test-slave-s9ct krb5kdc[20257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:23:26 dist-test-slave-s9ct krb5kdc[20257](info): set up 2 sockets
Sep 29 16:23:26 dist-test-slave-s9ct krb5kdc[20257](info): commencing operation
krb5kdc: starting...
W20260929 16:23:27.070832 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:23:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:23:27.197521 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:27.308069 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:27.308344 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:27.308419 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:27.312414 20270 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:23:27.312461 20270 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:23:27.312477 20270 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:23:27.312491 20270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:27.312508 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:27.312526 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:27.312582 20270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:27.312608 20270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:27.317806 20270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:27.318873 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:27.319892 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:27.328851 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.328922 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.328990 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:27.328908 20270 server_base.cc:1034] running on GCE node
I20260929 16:23:27.329648 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:27.330832 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:27.332026 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790699007332003 us; error 44 us; skew 500 ppm
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:27.335026 20270 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:27.335968 20270 webserver.cc:460] Webserver started at http://127.4.200.126:34307/ using document root <none> and password file <none>
I20260929 16:23:27.336566 20270 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:27.336659 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:27.336901 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:27.339073 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "b51fa78f3bf642a7a348f59142bd6362"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "14b9d1681e4345873b2e0605e4c6cf7a"
server_key_iv: "fc85058f870bf9a371b3b387dc79f0b8"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.339653 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "b51fa78f3bf642a7a348f59142bd6362"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "14b9d1681e4345873b2e0605e4c6cf7a"
server_key_iv: "fc85058f870bf9a371b3b387dc79f0b8"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.343518 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:23:27.345999 20285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.347404 20270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:23:27.347555 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b51fa78f3bf642a7a348f59142bd6362"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "14b9d1681e4345873b2e0605e4c6cf7a"
server_key_iv: "fc85058f870bf9a371b3b387dc79f0b8"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.347677 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:27.363710 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:27.366955 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:27.367172 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:27.374989 20270 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45181
I20260929 16:23:27.375317 20337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45181 every 8 connection(s)
I20260929 16:23:27.376039 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:27.378865 20338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:27.383980 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20270
I20260929 16:23:27.384132 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:23:27.384408 20338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap starting.
I20260929 16:23:27.384459 4897 external_mini_cluster.cc:1428] Setting key 3e93fb4234696fad11042c2fceece550
I20260929 16:23:27.386914 20338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:27.387697 20338 log.cc:824] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:27.389573 20338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: No bootstrap required, opened a new log
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:27.393956 20338 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:27.394196 20338 raft_consensus.cc:374] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:27.394284 20338 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Initialized, Role: FOLLOWER
I20260929 16:23:27.394845 20338 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:27.395009 20338 raft_consensus.cc:388] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:27.395094 20338 raft_consensus.cc:482] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:27.395201 20338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:27.396067 20338 raft_consensus.cc:504] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:27.396409 20338 leader_election.cc:302] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b51fa78f3bf642a7a348f59142bd6362; no voters:
I20260929 16:23:27.396731 20338 leader_election.cc:288] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:23:27.396796 20343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:27.397015 20343 raft_consensus.cc:686] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 LEADER]: Becoming Leader. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Running, Role: LEADER
I20260929 16:23:27.397361 20343 consensus_queue.cc:237] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:27.398015 20338 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:27.399109 20345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b51fa78f3bf642a7a348f59142bd6362. Latest consensus state: current_term: 1 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:23:27.399405 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:27.385856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.4.200.126:45181)
0929 16:23:27.386257 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:27.386282 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:27.386317 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:23:27.387079 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:27.387097 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:27.387428 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:27.387707 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:27.388598 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.389569 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.390744 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.391191 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.391317 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:27.394112 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:27.394117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:27.394129 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:27.394155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:27.395983 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.396437 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.396442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.396448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.396594 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.396917 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.396921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.396922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.397421 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:27.397604 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:27.397991 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:23:27.398342 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260929 16:23:27.399534 20345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:27.399400 20344 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:23:27.400012 20344 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
W20260929 16:23:27.403391 20359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:23:27.403477 20359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:23:27.403555 20353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:27.404211 20353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:27.409407 20353 catalog_manager.cc:1347] Generated new cluster ID: 377b59c3ca0a4bee809a927619c34562
I20260929 16:23:27.409469 20353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:27.426316 20353 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:23:27.427277 20353 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:27.432679 20353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Generated new TSK 0
I20260929 16:23:27.433364 20353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:23:27.465879 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:27.579170 20364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:27.579460 20364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:27.579538 20364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:27.583540 20364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:27.583629 20364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:27.583719 20364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:23:27.588902 20364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:27.589936 20364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:27.590971 20364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:27.600530 20369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.600548 20370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.600522 20372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:27.600850 20364 server_base.cc:1034] running on GCE node
I20260929 16:23:27.601243 20364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:27.601799 20364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:27.602955 20364 hybrid_clock.cc:648] HybridClock initialized: now 1790699007602942 us; error 23 us; skew 500 ppm
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:27.605502 20364 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:23:27.606310 20364 webserver.cc:460] Webserver started at http://127.4.200.65:41371/ using document root <none> and password file <none>
I20260929 16:23:27.606870 20364 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:27.606938 20364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:27.607093 20364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:27.608966 20364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "b5183d72c8d84812b3d022e575c52e4b"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "3438906f4469ed8075681cdf30ffce31"
server_key_iv: "990567efaa1111a9226bac2029dc070d"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.609439 20364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "b5183d72c8d84812b3d022e575c52e4b"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "3438906f4469ed8075681cdf30ffce31"
server_key_iv: "990567efaa1111a9226bac2029dc070d"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.612921 20364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:23:27.615242 20379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.616406 20364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:23:27.616539 20364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "b5183d72c8d84812b3d022e575c52e4b"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "3438906f4469ed8075681cdf30ffce31"
server_key_iv: "990567efaa1111a9226bac2029dc070d"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.616648 20364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:27.633895 20364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:27.636734 20364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:27.636948 20364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:27.637521 20364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:27.638331 20364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:27.638408 20364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.638504 20364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:27.638535 20364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.647746 20364 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:39755
I20260929 16:23:27.647779 20492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:39755 every 8 connection(s)
I20260929 16:23:27.648581 20364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:23:27.652493 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20364
I20260929 16:23:27.652626 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:23:27.652985 4897 external_mini_cluster.cc:1428] Setting key 1e12ba456e43c7aa5f4236f51ad5e41b
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:23:27.662652 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:27.650279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:38391 (local address 127.4.200.126:45181)
0929 16:23:27.650450 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:27.650454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:27.651269 (+ 815us) server_negotiation.cc:407] Connection header received
0929 16:23:27.652730 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:27.652733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:27.652780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:27.652886 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:27.654450 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.655231 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.656493 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.656745 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.656856 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:27.659218 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:27.659221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:27.659223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:27.659244 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:27.660644 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.661134 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.661139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.661141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.661221 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.661556 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.661561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.661563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.661734 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:27.661840 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:27.662342 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:23:27.662481 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:23:27.663491 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:27.650706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:38391)
0929 16:23:27.651125 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:27.651164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:27.652491 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:27.653107 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:27.653121 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:27.653623 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:27.654268 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:27.654290 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.655400 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.655403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:27.656347 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:27.656357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.656891 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.656895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:27.657069 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:27.657642 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:27.657656 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:27.659058 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:27.660762 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:27.660767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:27.660780 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:27.661031 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:27.661313 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:27.661317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:27.661320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:27.661460 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:27.661838 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:27.661845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:27.662111 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:23:27.662302 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 16:23:27.662649 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:23:27.664621 20493 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:27.664862 20493 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:27.665331 20493 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:27.666781 20302 ts_manager.cc:194] Registered new tserver with Master: b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:27.667888 20302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:38391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:23:27.680542 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:27.788285 20498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:27.788534 20498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:27.788576 20498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:27.792531 20498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:27.792589 20498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:27.792686 20498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:23:27.797832 20498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:27.798915 20498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:27.799971 20498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:27.809268 20503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.809333 20504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:27.809356 20506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:27.809619 20498 server_base.cc:1034] running on GCE node
I20260929 16:23:27.810063 20498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:27.810751 20498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:27.811954 20498 hybrid_clock.cc:648] HybridClock initialized: now 1790699007811944 us; error 30 us; skew 500 ppm
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:27.814640 20498 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:23:27.815536 20498 webserver.cc:460] Webserver started at http://127.4.200.66:37223/ using document root <none> and password file <none>
I20260929 16:23:27.816094 20498 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:27.816217 20498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:27.816442 20498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:27.818470 20498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "e73f9c299a2742a7ac8559197319bd6c"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "96fb41e9192824b64bdbf16c436f370c"
server_key_iv: "7a52c1727421d97784ab7c4334f12482"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.819018 20498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "e73f9c299a2742a7ac8559197319bd6c"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "96fb41e9192824b64bdbf16c436f370c"
server_key_iv: "7a52c1727421d97784ab7c4334f12482"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.822746 20498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:23:27.824959 20513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.826092 20498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:27.826267 20498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "e73f9c299a2742a7ac8559197319bd6c"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "96fb41e9192824b64bdbf16c436f370c"
server_key_iv: "7a52c1727421d97784ab7c4334f12482"
server_key_version: "encryptionkey@0"
I20260929 16:23:27.826378 20498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:27.843667 20498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:27.846580 20498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:27.846776 20498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:27.847363 20498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:27.848165 20498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:27.848237 20498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.848304 20498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:27.848357 20498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:27.857350 20498 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38027
I20260929 16:23:27.857378 20626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38027 every 8 connection(s)
I20260929 16:23:27.858207 20498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:23:27.866101 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20498
I20260929 16:23:27.866247 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:23:27.866595 4897 external_mini_cluster.cc:1428] Setting key bcd16bc333020e9c61f1db4669451d26
Sep 29 16:23:27 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:27.871786 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:27.860040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:51491 (local address 127.4.200.126:45181)
0929 16:23:27.860177 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:27.860180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:27.860955 (+ 775us) server_negotiation.cc:407] Connection header received
0929 16:23:27.862250 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:27.862253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:27.862310 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:27.862410 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:27.863538 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.864079 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.865220 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.865454 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.865568 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:27.868336 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:27.868339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:27.868341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:27.868365 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:27.869765 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.870264 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.870270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.870271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.870352 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:27.870735 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:27.870738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:27.870740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:27.870923 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:27.871029 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:27.871502 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:23:27.871620 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:23:27.872627 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:27.860385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:51491)
0929 16:23:27.860804 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:27.860845 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:27.862041 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:27.862580 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:27.862588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:27.862929 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:27.863388 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:27.863401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.864224 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.864227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:27.865090 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:27.865100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:27.865604 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:27.865608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:27.865772 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:27.866447 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:27.866472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:27.868174 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:27.869883 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:27.869891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:27.869905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:27.870157 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:27.870486 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:27.870489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:27.870491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:27.870629 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:27.871027 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:27.871034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:27.871267 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 16:23:27.871460 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 16:23:27.871764 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260929 16:23:27.873770 20627 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:27.874011 20627 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:27.874537 20627 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:27.875571 20302 ts_manager.cc:194] Registered new tserver with Master: e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027)
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:23:27.876183 20302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:51491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:23:27.893688 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:28.003984 20632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:28.004226 20632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:28.004269 20632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:28.008262 20632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:28.008311 20632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:28.008368 20632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:23:28.013480 20632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:28.014451 20632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:28.015388 20632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:28.024690 20638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:28.024685 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:28.024788 20640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:28.025089 20632 server_base.cc:1034] running on GCE node
I20260929 16:23:28.025455 20632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:28.026027 20632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:28.027184 20632 hybrid_clock.cc:648] HybridClock initialized: now 1790699008027137 us; error 59 us; skew 500 ppm
Sep 29 16:23:28 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:28.030030 20632 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:23:28.030913 20632 webserver.cc:460] Webserver started at http://127.4.200.67:42947/ using document root <none> and password file <none>
I20260929 16:23:28.031455 20632 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:28.031545 20632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:28.031746 20632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:23:28.033649 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "96ed95ea51124535b59221a78bf3230b"
format_stamp: "Formatted at 2026-09-29 16:23:28 on dist-test-slave-s9ct"
server_key: "ee3154c12b2c34326253a899eea23a60"
server_key_iv: "e87dd8979942bff62fea0f619e106add"
server_key_version: "encryptionkey@0"
I20260929 16:23:28.034195 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "96ed95ea51124535b59221a78bf3230b"
format_stamp: "Formatted at 2026-09-29 16:23:28 on dist-test-slave-s9ct"
server_key: "ee3154c12b2c34326253a899eea23a60"
server_key_iv: "e87dd8979942bff62fea0f619e106add"
server_key_version: "encryptionkey@0"
I20260929 16:23:28.037830 20632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:23:28.040233 20647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:28.041361 20632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:28.041507 20632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "96ed95ea51124535b59221a78bf3230b"
format_stamp: "Formatted at 2026-09-29 16:23:28 on dist-test-slave-s9ct"
server_key: "ee3154c12b2c34326253a899eea23a60"
server_key_iv: "e87dd8979942bff62fea0f619e106add"
server_key_version: "encryptionkey@0"
I20260929 16:23:28.041615 20632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:28.076915 20632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:28.080065 20632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:28.080276 20632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:28.080899 20632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:28.081766 20632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:28.081838 20632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:28.081913 20632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:28.081966 20632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:28.091293 20632 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:33325
I20260929 16:23:28.091310 20760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:33325 every 8 connection(s)
I20260929 16:23:28.092219 20632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:23:28.100284 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20632
I20260929 16:23:28.100426 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:23:28.100740 4897 external_mini_cluster.cc:1428] Setting key c41b7eeb01061e18487982b3c488104a
Sep 29 16:23:28 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:28.107021 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.094080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47033 (local address 127.4.200.126:45181)
0929 16:23:28.094233 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.094238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.095206 (+ 968us) server_negotiation.cc:407] Connection header received
0929 16:23:28.096469 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.096472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.096521 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.096597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:28.097782 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.098532 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.100201 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.100540 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.100658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.103488 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:28.103492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:28.103494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:28.103520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:28.105135 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:28.105595 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:28.105597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:28.105598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:28.105671 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:28.105970 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:28.105972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:28.105973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:28.106166 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:28.106240 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.106799 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.106901 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:23:28.107919 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.094595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:47033)
0929 16:23:28.095046 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.095095 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.096261 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.096776 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.096783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.097157 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:28.097639 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.097653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.098664 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.098668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.100073 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.100084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.100722 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.100726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.100948 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.101619 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:28.101636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:28.103300 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:28.105234 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:28.105239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:28.105252 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:28.105512 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:28.105749 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:28.105751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:28.105752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:28.105894 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:28.106252 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:28.106281 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:28.106591 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.106754 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.107029 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260929 16:23:28.109208 20761 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:28.109436 20761 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:28.109894 20761 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:28.110946 20302 ts_manager.cc:194] Registered new tserver with Master: 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
I20260929 16:23:28.111532 20302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:47033
I20260929 16:23:28.114732 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:23:28.124487 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.116862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.4.200.126:45181)
0929 16:23:28.117020 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.117025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.117189 (+ 164us) server_negotiation.cc:407] Connection header received
0929 16:23:28.117352 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.117355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.117399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.117493 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:28.118186 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.118819 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.119686 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.120005 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.120098 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.121502 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:28.121506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:28.121509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:28.121534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:28.122788 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:28.123211 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:28.123215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:28.123217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:28.123310 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:28.123583 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:28.123588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:28.123589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:28.123817 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:28.123960 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.124190 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.124318 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:23:28.127398 20302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:23:28.130749 20302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:23:28.145604 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.139717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.4.200.66:38027)
0929 16:23:28.140090 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.140105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.140709 (+ 604us) server_negotiation.cc:407] Connection header received
0929 16:23:28.140998 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.141003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.141236 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.141403 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:28.141735 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.142714 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.144230 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.144803 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.144952 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.145111 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.145324 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.145445 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":182,"threads_started":1}
I20260929 16:23:28.146039 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.140443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.4.200.65:39755)
0929 16:23:28.140713 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.140726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.141268 (+ 542us) server_negotiation.cc:407] Connection header received
0929 16:23:28.141361 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.141366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.141530 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.141675 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:28.141951 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.143191 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.144572 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.145460 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.145577 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.145715 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.145808 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.145891 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
I20260929 16:23:28.146088 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.140318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.0.0.1:38376)
0929 16:23:28.141152 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.141164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.141256 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.141675 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.141678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.141695 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:28.141826 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.141833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.143354 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.143357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.144431 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.144442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.145580 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.145583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.145667 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.145688 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.145762 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.145891 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":556,"thread_start_us":108,"threads_started":1}
I20260929 16:23:28.146363 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.139545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.0.0.1:58538)
0929 16:23:28.140566 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.140619 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.140821 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.141403 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.141409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.141430 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:28.141596 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.141604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.142836 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.142841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.144111 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.144120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.144995 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.144999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.145105 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.145147 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.146124 (+ 977us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.146208 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":22,"thread_start_us":78,"threads_started":1}
I20260929 16:23:28.146850 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.139549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33325 (local address 127.0.0.1:59612)
0929 16:23:28.140862 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.140877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.141023 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.141715 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.141718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.141735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:28.141861 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.141867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.143354 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.143357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.145193 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.145206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.146427 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.146432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.146523 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.146555 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.146667 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.146734 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":1101,"thread_start_us":95,"threads_started":1}
I20260929 16:23:28.146984 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.139807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.4.200.67:33325)
0929 16:23:28.140536 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.140550 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.140956 (+ 406us) server_negotiation.cc:407] Connection header received
0929 16:23:28.141184 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.141193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.141412 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.141556 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:28.142005 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.143191 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.145389 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.146227 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.146325 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.146491 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.146713 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.146828 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":149,"threads_started":1}
I20260929 16:23:28.149197 20427 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d50052a4346a78fecbd9427b7905e (DEFAULT_TABLE table=test-table [id=b1331e87db824a8ea7810359713a2558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:28.149281 20695 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d50052a4346a78fecbd9427b7905e (DEFAULT_TABLE table=test-table [id=b1331e87db824a8ea7810359713a2558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:28.149281 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d50052a4346a78fecbd9427b7905e (DEFAULT_TABLE table=test-table [id=b1331e87db824a8ea7810359713a2558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:28.150213 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d50052a4346a78fecbd9427b7905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:28.150261 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d50052a4346a78fecbd9427b7905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:28.150267 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d50052a4346a78fecbd9427b7905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:28.156889 20778 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Bootstrap starting.
I20260929 16:23:28.158012 20779 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Bootstrap starting.
I20260929 16:23:28.158725 20780 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Bootstrap starting.
I20260929 16:23:28.159159 20778 tablet_bootstrap.cc:654] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:28.160221 20778 log.cc:824] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:28.160490 20779 tablet_bootstrap.cc:654] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:28.160763 20780 tablet_bootstrap.cc:654] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:28.161406 20779 log.cc:824] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:28.161926 20780 log.cc:824] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:28.162328 20778 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: No bootstrap required, opened a new log
I20260929 16:23:28.162562 20778 ts_tablet_manager.cc:1397] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:28.163443 20779 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: No bootstrap required, opened a new log
I20260929 16:23:28.163633 20779 ts_tablet_manager.cc:1397] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:28.163926 20780 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: No bootstrap required, opened a new log
I20260929 16:23:28.164136 20780 ts_tablet_manager.cc:1397] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:28.165690 20778 raft_consensus.cc:348] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.165941 20778 raft_consensus.cc:374] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:28.166016 20778 raft_consensus.cc:729] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5183d72c8d84812b3d022e575c52e4b, State: Initialized, Role: FOLLOWER
I20260929 16:23:28.166531 20778 consensus_queue.cc:260] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.167152 20493 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
I20260929 16:23:28.167484 20778 ts_tablet_manager.cc:1428] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:23:28.167869 20780 raft_consensus.cc:348] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.168155 20780 raft_consensus.cc:374] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:28.168260 20780 raft_consensus.cc:729] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ed95ea51124535b59221a78bf3230b, State: Initialized, Role: FOLLOWER
I20260929 16:23:28.168774 20780 consensus_queue.cc:260] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.168681 20779 raft_consensus.cc:348] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.168924 20779 raft_consensus.cc:374] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:28.169013 20779 raft_consensus.cc:729] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73f9c299a2742a7ac8559197319bd6c, State: Initialized, Role: FOLLOWER
I20260929 16:23:28.169490 20779 consensus_queue.cc:260] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.169821 20780 ts_tablet_manager.cc:1428] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:28.170167 20627 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
I20260929 16:23:28.170372 20779 ts_tablet_manager.cc:1428] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:23:28.170666 20761 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
I20260929 16:23:28.269153 20786 raft_consensus.cc:482] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:28.269428 20786 raft_consensus.cc:504] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.270637 20786 leader_election.cc:288] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325), b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:28.275712 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.270970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.4.200.66:36405)
0929 16:23:28.271115 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.271128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.271254 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.271627 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.271632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.271655 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:28.271790 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.271796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.272884 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.272886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.274225 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.274235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.275148 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.275151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.275304 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.275364 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.275444 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.275542 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":69}
I20260929 16:23:28.275732 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.271218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:36405 (local address 127.4.200.65:39755)
0929 16:23:28.271360 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.271366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.271385 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:23:28.271444 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.271447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.271511 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.271580 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:28.271942 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.272770 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.274369 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.275019 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.275147 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.275202 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.275507 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.275591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:23:28.275724 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.271117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:40275 (local address 127.4.200.67:33325)
0929 16:23:28.271257 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.271262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.271684 (+ 422us) server_negotiation.cc:407] Connection header received
0929 16:23:28.271747 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.271750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.271786 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.271852 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:28.272115 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.272848 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.274369 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.275018 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.275147 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.275207 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.275507 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.275591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:23:28.275722 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.270970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33325 (local address 127.4.200.66:40275)
0929 16:23:28.271550 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.271583 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:28.271674 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:28.271880 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:28.271883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:28.271903 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:28.272011 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.272015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.272946 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.272949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.274228 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:28.274235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.275148 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.275151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:28.275307 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.275360 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:23:28.275444 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:23:28.275541 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":324,"thread_start_us":152,"threads_started":1}
I20260929 16:23:28.276605 20715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d50052a4346a78fecbd9427b7905e" candidate_uuid: "e73f9c299a2742a7ac8559197319bd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ed95ea51124535b59221a78bf3230b" is_pre_election: true
I20260929 16:23:28.276607 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d50052a4346a78fecbd9427b7905e" candidate_uuid: "e73f9c299a2742a7ac8559197319bd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5183d72c8d84812b3d022e575c52e4b" is_pre_election: true
I20260929 16:23:28.276942 20715 raft_consensus.cc:2454] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73f9c299a2742a7ac8559197319bd6c in term 0.
I20260929 16:23:28.276942 20447 raft_consensus.cc:2454] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e73f9c299a2742a7ac8559197319bd6c in term 0.
I20260929 16:23:28.277458 20515 leader_election.cc:302] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c; no voters:
I20260929 16:23:28.277693 20786 raft_consensus.cc:2781] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:28.277792 20786 raft_consensus.cc:482] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:28.277868 20786 raft_consensus.cc:3037] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:28.279009 20786 raft_consensus.cc:504] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.279435 20786 leader_election.cc:288] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [CANDIDATE]: Term 1 election: Requested vote from peers 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325), b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:28.279858 20715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d50052a4346a78fecbd9427b7905e" candidate_uuid: "e73f9c299a2742a7ac8559197319bd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ed95ea51124535b59221a78bf3230b"
I20260929 16:23:28.279858 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d50052a4346a78fecbd9427b7905e" candidate_uuid: "e73f9c299a2742a7ac8559197319bd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5183d72c8d84812b3d022e575c52e4b"
I20260929 16:23:28.280002 20447 raft_consensus.cc:3037] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:28.280002 20715 raft_consensus.cc:3037] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:28.281333 20715 raft_consensus.cc:2454] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e73f9c299a2742a7ac8559197319bd6c in term 1.
I20260929 16:23:28.281349 20447 raft_consensus.cc:2454] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e73f9c299a2742a7ac8559197319bd6c in term 1.
I20260929 16:23:28.281692 20517 leader_election.cc:302] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b, e73f9c299a2742a7ac8559197319bd6c; no voters:
I20260929 16:23:28.281888 20786 raft_consensus.cc:2781] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:28.282137 20786 raft_consensus.cc:686] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 1 LEADER]: Becoming Leader. State: Replica: e73f9c299a2742a7ac8559197319bd6c, State: Running, Role: LEADER
I20260929 16:23:28.282541 20786 consensus_queue.cc:237] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:28.285745 20300 catalog_manager.cc:5495] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c reported cstate change: term changed from 0 to 1, leader changed from <none> to e73f9c299a2742a7ac8559197319bd6c (127.4.200.66). New cstate: current_term: 1 leader_uuid: "e73f9c299a2742a7ac8559197319bd6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260929 16:23:28.298126 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.294662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.4.200.66:38027)
0929 16:23:28.294793 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:28.294797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:28.294811 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:23:28.294951 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:28.294954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:28.295008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:28.295099 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:28.295356 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.295877 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.296693 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:28.296918 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:28.297035 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:28.297291 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:28.297749 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:28.297851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:28.297923 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 16:23:28.297981 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:23:28.304800 20447 raft_consensus.cc:1261] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Refusing update from remote peer e73f9c299a2742a7ac8559197319bd6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:28.304800 20715 raft_consensus.cc:1261] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Refusing update from remote peer e73f9c299a2742a7ac8559197319bd6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:28.305559 20788 consensus_queue.cc:1035] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:28.305775 20786 consensus_queue.cc:1035] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:28.314169 20793 mvcc.cc:205] Tried to move back new op lower bound from 7334703138008256512 to 7334703137927872512. Current Snapshot: MvccSnapshot[applied={T|T < 7334703138008256512}]
I20260929 16:23:28.315192 20792 mvcc.cc:205] Tried to move back new op lower bound from 7334703138008256512 to 7334703137927872512. Current Snapshot: MvccSnapshot[applied={T|T < 7334703138008256512}]
I20260929 16:23:28.322880 20300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:28.323081 20300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:28.325766 20300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Sending DeleteTablet for 3 replicas of tablet 6b0d50052a4346a78fecbd9427b7905e
I20260929 16:23:28.326674 20561 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d50052a4346a78fecbd9427b7905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:58538
I20260929 16:23:28.326673 20695 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d50052a4346a78fecbd9427b7905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:59612
I20260929 16:23:28.326711 20427 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d50052a4346a78fecbd9427b7905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:38376
I20260929 16:23:28.327190 20802 tablet_replica.cc:330] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: stopping tablet replica
I20260929 16:23:28.327399 20803 tablet_replica.cc:330] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: stopping tablet replica
I20260929 16:23:28.327435 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20364
I20260929 16:23:28.327580 20802 raft_consensus.cc:2229] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:28.328380 20802 raft_consensus.cc:2258] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:28.331893 20802 ts_tablet_manager.cc:1905] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:28.334640 20801 tablet_replica.cc:330] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: stopping tablet replica
I20260929 16:23:28.335043 20801 raft_consensus.cc:2229] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:23:28.335470 20802 ts_tablet_manager.cc:1918] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:28.335582 20801 raft_consensus.cc:2258] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:28.335640 20802 log.cc:1196] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/6b0d50052a4346a78fecbd9427b7905e
I20260929 16:23:28.336210 20802 ts_tablet_manager.cc:1939] T 6b0d50052a4346a78fecbd9427b7905e P 96ed95ea51124535b59221a78bf3230b: Deleting consensus metadata
I20260929 16:23:28.337180 20801 ts_tablet_manager.cc:1905] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:23:28.337376 20289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:39755 (error 108)
I20260929 16:23:28.337633 20287 catalog_manager.cc:4841] TS 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): tablet 6b0d50052a4346a78fecbd9427b7905e (table test-table [id=b1331e87db824a8ea7810359713a2558]) successfully deleted
I20260929 16:23:28.338559 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20498
I20260929 16:23:28.338958 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.338564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.0.0.1:38382)
0929 16:23:28.338777 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.338853 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 16:23:28.340240 20289 catalog_manager.cc:4584] TS b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0d50052a4346a78fecbd9427b7905e: Network error: Client connection negotiation failed: client connection to 127.4.200.65:39755: connect: Connection refused (error 111)
I20260929 16:23:28.344730 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20632
I20260929 16:23:28.345649 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:28.344973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.0.0.1:58568)
0929 16:23:28.345532 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:28.345579 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.66:38027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260929 16:23:28.350382 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20270
I20260929 16:23:48.356949 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--webserver_interface=127.4.200.126
--webserver_port=34307
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:23:48.471432 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:48.471696 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:48.471736 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:48.475946 20805 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:23:48.476002 20805 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:23:48.476018 20805 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:23:48.476032 20805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:23:48.476050 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:48.476107 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:48.476137 20805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:23:48.476169 20805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:23:48.481621 20805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:48.482784 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:48.483806 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:23:48.497519 20805 server_base.cc:1034] running on GCE node
W20260929 16:23:48.497430 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:48.497421 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:48.497536 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:48.498461 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:48.499722 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:48.500931 20805 hybrid_clock.cc:648] HybridClock initialized: now 1790699028500890 us; error 56 us; skew 500 ppm
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:48.511485 20805 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:23:48.512440 20805 webserver.cc:460] Webserver started at http://127.4.200.126:34307/ using document root <none> and password file <none>
I20260929 16:23:48.513006 20805 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:48.513093 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:48.517333 20805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:23:48.519583 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:48.520844 20805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:23:48.520989 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b51fa78f3bf642a7a348f59142bd6362"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "14b9d1681e4345873b2e0605e4c6cf7a"
server_key_iv: "fc85058f870bf9a371b3b387dc79f0b8"
server_key_version: "encryptionkey@0"
I20260929 16:23:48.521593 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:48.536398 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:48.539865 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:48.540081 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:48.547482 20805 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45181
I20260929 16:23:48.547816 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45181 every 8 connection(s)
I20260929 16:23:48.548640 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:23:48.553207 20874 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:23:48.554039 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20805
I20260929 16:23:48.554553 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:39755
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=41371
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:48.555429 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap starting.
I20260929 16:23:48.572887 20874 log.cc:824] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:48.581749 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:48.582185 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap complete.
I20260929 16:23:48.586337 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:48.586838 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Initialized, Role: FOLLOWER
I20260929 16:23:48.587385 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:48.587510 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:23:48.587603 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:23:48.587718 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:23:48.589759 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:48.590106 20874 leader_election.cc:302] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b51fa78f3bf642a7a348f59142bd6362; no voters:
I20260929 16:23:48.590462 20874 leader_election.cc:288] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:23:48.590577 20878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:23:48.590839 20878 raft_consensus.cc:686] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 LEADER]: Becoming Leader. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Running, Role: LEADER
I20260929 16:23:48.591210 20878 consensus_queue.cc:237] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:23:48.591686 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:23:48.592721 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:23:48.592859 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:48.593225 20880 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b51fa78f3bf642a7a348f59142bd6362. Latest consensus state: current_term: 2 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:23:48.593395 20880 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
I20260929 16:23:48.593715 20887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:23:48.598676 20887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:23:48.599486 20887 catalog_manager.cc:1259] Loaded cluster ID: 377b59c3ca0a4bee809a927619c34562
I20260929 16:23:48.599556 20887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:23:48.602121 20887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:23:48.602905 20887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Loaded TSK: 0
I20260929 16:23:48.621982 20887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Generated new TSK 1
I20260929 16:23:48.622275 20887 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:23:48.676064 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:48.676386 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:48.676458 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:48.680796 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:48.680891 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:48.680980 20876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:23:48.686542 20876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=41371
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:48.687672 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:48.688741 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:48.702342 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:48.703027 20876 server_base.cc:1034] running on GCE node
W20260929 16:23:48.702318 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:48.703378 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:48.703888 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:48.706352 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:48.707664 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790699028707639 us; error 53 us; skew 500 ppm
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:48.711117 20876 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:23:48.712141 20876 webserver.cc:460] Webserver started at http://127.4.200.65:41371/ using document root <none> and password file <none>
I20260929 16:23:48.712754 20876 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:48.712867 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:48.717293 20876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:23:48.719720 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:48.721086 20876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:23:48.721231 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "b5183d72c8d84812b3d022e575c52e4b"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "3438906f4469ed8075681cdf30ffce31"
server_key_iv: "990567efaa1111a9226bac2029dc070d"
server_key_version: "encryptionkey@0"
I20260929 16:23:48.721853 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:48.734699 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:48.738076 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:48.738289 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:48.738920 20876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:48.740109 20918 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:23:48.745428 20876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:23:48.745513 20876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 16:23:48.745621 20876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:23:48.747346 20876 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:23:48.747416 20876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260929 16:23:48.747710 20918 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Bootstrap starting.
I20260929 16:23:48.757920 20876 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:39755
I20260929 16:23:48.758019 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:39755 every 8 connection(s)
I20260929 16:23:48.759022 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:23:48.762089 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 20876
I20260929 16:23:48.762602 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:38027
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=37223
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:23:48.767601 20918 log.cc:824] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:48.776024 20918 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:23:48.776556 20918 tablet_bootstrap.cc:492] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Bootstrap complete.
I20260929 16:23:48.777961 20918 ts_tablet_manager.cc:1397] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.009s
I20260929 16:23:48.779795 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.761124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:39199 (local address 127.4.200.126:45181)
0929 16:23:48.761616 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:48.761636 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:48.761770 (+ 134us) server_negotiation.cc:407] Connection header received
0929 16:23:48.763385 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:48.763405 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:48.763827 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:48.764237 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:48.765658 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.767140 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.769114 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.769823 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.770013 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:48.773456 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:48.773467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:48.773491 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:48.773538 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:48.775984 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:48.776738 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:48.776745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:48.776751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:48.776920 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:48.777332 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:48.777339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:48.777352 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:48.777849 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:48.778093 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:48.778661 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 16:23:48.778970 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":115,"threads_started":1}
I20260929 16:23:48.779815 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.761021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:39199)
0929 16:23:48.761584 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:48.761642 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:48.762881 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:48.764628 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:48.764634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:48.764996 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:48.765468 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.765483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.767335 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.767342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.768949 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.768965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.770008 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.770015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.770275 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:48.771083 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:48.771110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:48.773141 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:48.776134 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:48.776142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:48.776155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:48.776549 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:48.777033 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:48.777036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:48.777038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:48.777224 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:48.777976 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:48.777982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:48.778281 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 16:23:48.778517 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 16:23:48.778871 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260929 16:23:48.781034 20918 raft_consensus.cc:348] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:48.781466 20918 raft_consensus.cc:729] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5183d72c8d84812b3d022e575c52e4b, State: Initialized, Role: FOLLOWER
I20260929 16:23:48.781980 20918 consensus_queue.cc:260] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:48.782902 21026 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:48.783105 20918 ts_tablet_manager.cc:1428] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:23:48.783129 21026 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:48.783721 21026 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:48.786574 20838 ts_manager.cc:194] Registered new tserver with Master: b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:48.791092 20838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:39199
I20260929 16:23:48.794463 21026 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.65@KRBTEST.COM
I20260929 16:23:48.803570 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.790093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.0.0.1:49552)
0929 16:23:48.790556 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:48.790609 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:48.790750 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:48.791667 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:48.791673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:48.792130 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:48.792295 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.792306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.793369 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.793376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.794102 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.794114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.795054 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.795058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.795202 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:48.795942 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:48.795961 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:48.798179 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:48.801087 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:48.801100 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:48.801106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:48.801594 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:48.802010 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:48.802013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:48.802015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:48.802141 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:48.802848 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:48.802853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:48.803006 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 16:23:48.803154 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:23:48.803374 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":154,"threads_started":1}
I20260929 16:23:48.804301 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.790161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.4.200.65:39755)
0929 16:23:48.790975 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:48.790986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:48.791007 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:23:48.791070 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:48.791074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:48.791312 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:48.791473 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:48.792461 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.793223 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.794249 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.794903 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.795029 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:48.798445 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:48.798456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:48.798472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:48.798502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:48.800942 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:48.801762 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:48.801769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:48.801771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:48.801888 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:48.802274 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:48.802277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:48.802281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:48.802700 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:48.802853 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:48.803313 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 16:23:48.803543 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":125,"threads_started":1}
I20260929 16:23:48.805356 20960 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d50052a4346a78fecbd9427b7905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:28 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:49552
I20260929 16:23:48.806519 21035 tablet_replica.cc:330] stopping tablet replica
I20260929 16:23:48.806821 21035 raft_consensus.cc:2229] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:48.807081 21035 raft_consensus.cc:2258] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:48.808050 21035 ts_tablet_manager.cc:1905] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:48.811785 21035 ts_tablet_manager.cc:1918] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:48.811906 21035 log.cc:1196] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal/wals/6b0d50052a4346a78fecbd9427b7905e
I20260929 16:23:48.812191 21035 ts_tablet_manager.cc:1939] T 6b0d50052a4346a78fecbd9427b7905e P b5183d72c8d84812b3d022e575c52e4b: Deleting consensus metadata
I20260929 16:23:48.813972 20825 catalog_manager.cc:4841] TS b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755): tablet 6b0d50052a4346a78fecbd9427b7905e (table test-table [id=b1331e87db824a8ea7810359713a2558]) successfully deleted
W20260929 16:23:48.893146 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:48.893426 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:48.893473 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:48.898957 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:48.899039 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:48.899104 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:23:48.904754 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=37223
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:48.905869 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:48.906955 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:48.916564 21041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:48.916580 21030 server_base.cc:1034] running on GCE node
W20260929 16:23:48.916615 21040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:48.916637 21043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:48.917276 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:48.917948 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:48.919116 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790699028919094 us; error 37 us; skew 500 ppm
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:48.921798 21030 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:23:48.922783 21030 webserver.cc:460] Webserver started at http://127.4.200.66:37223/ using document root <none> and password file <none>
I20260929 16:23:48.923341 21030 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:48.923415 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:48.927747 21030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:23:48.930015 21050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:48.931324 21030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:23:48.931461 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "e73f9c299a2742a7ac8559197319bd6c"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "96fb41e9192824b64bdbf16c436f370c"
server_key_iv: "7a52c1727421d97784ab7c4334f12482"
server_key_version: "encryptionkey@0"
I20260929 16:23:48.932018 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:48.964195 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:48.967522 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:48.967747 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:48.968384 21030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:48.969583 21057 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 16:23:48.973469 21057 ts_tablet_manager.cc:1852] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 16:23:48.973862 21057 ts_tablet_manager.cc:1905] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:48.978060 21057 ts_tablet_manager.cc:1918] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:48.978168 21057 log.cc:1196] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/6b0d50052a4346a78fecbd9427b7905e
I20260929 16:23:48.978569 21057 ts_tablet_manager.cc:1939] T 6b0d50052a4346a78fecbd9427b7905e P e73f9c299a2742a7ac8559197319bd6c: Deleting consensus metadata
I20260929 16:23:48.978894 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 16:23:48.978972 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260929 16:23:48.979075 21030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:48.979123 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:48.988257 21030 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38027
I20260929 16:23:48.988287 21164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38027 every 8 connection(s)
I20260929 16:23:48.989192 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:23:48.991776 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21030
I20260929 16:23:48.992291 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:33325
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=42947
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:23:48 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:49.006999 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.991027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:52545 (local address 127.4.200.126:45181)
0929 16:23:48.991183 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:48.991189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:48.992290 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 16:23:48.993778 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:48.993783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:48.993878 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:48.994015 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:48.995389 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.996322 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.998519 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.998972 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.999227 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.002787 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:49.002790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:49.002792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:49.002835 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:49.004516 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.005210 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.005213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.005214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.005298 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.005723 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.005727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.005728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.005946 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:49.006032 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.006717 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.006860 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:23:49.008049 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:48.991522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:52545)
0929 16:23:48.992098 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:48.992160 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:48.993497 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:48.994216 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:48.994224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:48.994670 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:48.995202 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.995217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.996601 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.996607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.998272 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:48.998293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:48.999283 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:48.999290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:48.999560 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.000355 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:49.000385 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:49.002545 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:49.004647 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:49.004654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:49.004674 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:49.005069 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:49.005391 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:49.005395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:49.005397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:49.005585 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:49.006053 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:49.006060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:49.006430 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.006643 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.007010 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":168,"threads_started":1}
I20260929 16:23:49.009457 21165 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:49.009722 21165 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:49.010244 21165 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:49.011457 20838 ts_manager.cc:194] Registered new tserver with Master: e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027)
I20260929 16:23:49.012235 20838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:52545
W20260929 16:23:49.113936 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:23:49.114202 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:23:49.114246 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:23:49.118556 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:23:49.118630 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:23:49.118693 21168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:23:49.124152 21168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:33325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=42947
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:23:49.125355 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:23:49.126592 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:23:49.136547 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:49.136571 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:23:49.136761 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:23:49.136940 21168 server_base.cc:1034] running on GCE node
I20260929 16:23:49.137367 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:23:49.138060 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:23:49.139251 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790699029139188 us; error 78 us; skew 500 ppm
Sep 29 16:23:49 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:23:49.142591 21168 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:23:49.143541 21168 webserver.cc:460] Webserver started at http://127.4.200.67:42947/ using document root <none> and password file <none>
I20260929 16:23:49.144130 21168 fs_manager.cc:359] Metadata directory not provided
I20260929 16:23:49.144235 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:23:49.148263 21168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:23:49.150484 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:49.151757 21168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:23:49.151901 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "96ed95ea51124535b59221a78bf3230b"
format_stamp: "Formatted at 2026-09-29 16:23:28 on dist-test-slave-s9ct"
server_key: "ee3154c12b2c34326253a899eea23a60"
server_key_iv: "e87dd8979942bff62fea0f619e106add"
server_key_version: "encryptionkey@0"
I20260929 16:23:49.152483 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:23:49.170284 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:23:49.173818 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:23:49.174060 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:23:49.174719 21168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:23:49.175582 21168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:23:49.175654 21168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:49.175729 21168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:23:49.175772 21168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:23:49.185057 21168 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:33325
I20260929 16:23:49.185073 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:33325 every 8 connection(s)
I20260929 16:23:49.186091 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:23:49.190701 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21168
Sep 29 16:23:49 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:23:49.200402 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.191387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.4.200.126:45181)
0929 16:23:49.191738 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.191742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.191772 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:23:49.191859 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.191862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.191909 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.191987 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:49.192901 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.193679 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.194837 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.195158 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.195284 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.196839 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:49.196843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:49.196848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:49.196878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:49.198637 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.199139 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.199142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.199143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.199218 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.199514 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.199516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.199517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.199687 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:49.199937 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.200069 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.200195 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260929 16:23:49.201759 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.187941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:34179 (local address 127.4.200.126:45181)
0929 16:23:49.188081 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.188085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.188870 (+ 785us) server_negotiation.cc:407] Connection header received
0929 16:23:49.190168 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.190171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.190231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.190336 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:49.191913 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.192781 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.194275 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.194658 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.194785 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.198035 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:49.198038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:49.198040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:49.198074 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:49.199628 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.200140 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.200145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.200147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.200227 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.200614 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.200617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.200619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.200813 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:49.200924 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.201531 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.201652 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:23:49.202857 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.188260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:34179)
0929 16:23:49.188710 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.188751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:49.189951 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:49.190560 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:49.190568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:49.190983 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:23:49.191741 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.191763 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.192908 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.192911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.194139 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.194149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.194894 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.194899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.195098 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.195848 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:23:49.195872 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:23:49.197842 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:23:49.199737 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:49.199743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:49.199761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:49.200029 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:49.200319 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:23:49.200323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:23:49.200325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:23:49.200516 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:23:49.200922 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:23:49.200930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:23:49.201259 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.201477 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.201843 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260929 16:23:49.203961 21297 heartbeater.cc:347] Connected to a master server at 127.4.200.126:45181
I20260929 16:23:49.204237 21297 heartbeater.cc:464] Registering TS with master...
I20260929 16:23:49.204764 21297 heartbeater.cc:511] Master 127.4.200.126:45181 requested a full tablet report, sending...
I20260929 16:23:49.205859 20838 ts_manager.cc:194] Registered new tserver with Master: 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
I20260929 16:23:49.206538 20838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:34179
I20260929 16:23:49.214360 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:23:49.218876 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.215133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.4.200.126:45181)
0929 16:23:49.215266 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.215271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.215500 (+ 229us) server_negotiation.cc:407] Connection header received
0929 16:23:49.215707 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.215712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.215825 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.215953 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:49.216208 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.216729 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.217597 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.217834 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.217946 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.218258 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:49.218441 (+ 183us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:23:49.218745 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39664: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:23:49.218964 21028 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39664: token expired
W20260929 16:23:49.219307 20764 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.200.126:45181)
I20260929 16:23:49.226666 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.219700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.4.200.126:45181)
0929 16:23:49.219821 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.219830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.219842 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:23:49.219936 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.219938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.219970 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.220031 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:23:49.220606 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.221079 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.221814 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.222004 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.222087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.223363 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:23:49.223366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:23:49.223368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:23:49.223391 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:23:49.225003 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.225444 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.225447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.225448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.225522 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:23:49.225807 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:23:49.225809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:23:49.225810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:23:49.226056 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:23:49.226181 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.226459 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.226548 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:23:49.229322 20838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:23:49.232659 20838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:23:49.240491 20960 tablet_service.cc:1467] Processing CreateTablet for tablet ee834331953a4a1fbfa736f8f92b151d (DEFAULT_TABLE table=test-table [id=c7a446002a054a67923356d0ff336f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:49.241058 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee834331953a4a1fbfa736f8f92b151d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:49.243685 20918 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Bootstrap starting.
I20260929 16:23:49.244043 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.239331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.0.0.1:43600)
0929 16:23:49.239477 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.239494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:49.239572 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:49.240466 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:49.240469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:49.240489 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:49.240593 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.240599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.241639 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.241644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.242790 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.242799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.243603 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.243607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.243699 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.243740 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.243818 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.243903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:23:49.244293 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.239558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.4.200.66:38027)
0929 16:23:49.239931 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.239944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.239969 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:23:49.240054 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.240059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.240285 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.240444 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:49.240688 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.241525 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.242920 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.243477 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.243603 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.243749 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.244016 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.244116 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260929 16:23:49.244647 20918 tablet_bootstrap.cc:654] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:49.246213 20918 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: No bootstrap required, opened a new log
I20260929 16:23:49.246328 20918 ts_tablet_manager.cc:1397] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:23:49.245993 21099 tablet_service.cc:1467] Processing CreateTablet for tablet ee834331953a4a1fbfa736f8f92b151d (DEFAULT_TABLE table=test-table [id=c7a446002a054a67923356d0ff336f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:49.246876 20918 raft_consensus.cc:348] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.246973 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee834331953a4a1fbfa736f8f92b151d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:49.246986 20918 raft_consensus.cc:374] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:49.247021 20918 raft_consensus.cc:729] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5183d72c8d84812b3d022e575c52e4b, State: Initialized, Role: FOLLOWER
I20260929 16:23:49.247141 20918 consensus_queue.cc:260] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.247380 20918 ts_tablet_manager.cc:1428] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:23:49.249580 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.239693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33325 (local address 127.0.0.1:51362)
0929 16:23:49.243983 (+ 4290us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.243998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:49.244092 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:49.244879 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:49.244883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:49.244903 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:49.245029 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.245037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.246373 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.246376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.247859 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.247871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.249138 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.249142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.249237 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.249262 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.249335 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.249441 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4226,"thread_start_us":94,"threads_started":1}
I20260929 16:23:49.250037 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.240101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.4.200.67:33325)
0929 16:23:49.240453 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.240463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.244198 (+ 3735us) server_negotiation.cc:407] Connection header received
0929 16:23:49.244285 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.244290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.244499 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.244666 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:49.245184 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.246206 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.247986 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.248795 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.248909 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.249070 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.249799 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.249904 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":133,"threads_started":1}
I20260929 16:23:49.252076 21057 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Bootstrap starting.
I20260929 16:23:49.252839 21231 tablet_service.cc:1467] Processing CreateTablet for tablet ee834331953a4a1fbfa736f8f92b151d (DEFAULT_TABLE table=test-table [id=c7a446002a054a67923356d0ff336f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:23:49.253923 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee834331953a4a1fbfa736f8f92b151d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:23:49.254283 21057 tablet_bootstrap.cc:654] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:49.255026 21057 log.cc:824] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:49.256778 21057 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: No bootstrap required, opened a new log
I20260929 16:23:49.256906 21057 ts_tablet_manager.cc:1397] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 16:23:49.259981 21057 raft_consensus.cc:348] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.260217 21057 raft_consensus.cc:374] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260929 16:23:49.260251 21027 tablet.cc:2369] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:49.260303 21057 raft_consensus.cc:729] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73f9c299a2742a7ac8559197319bd6c, State: Initialized, Role: FOLLOWER
I20260929 16:23:49.260779 21057 consensus_queue.cc:260] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.261175 21311 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Bootstrap starting.
I20260929 16:23:49.261466 21165 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
I20260929 16:23:49.261794 21057 ts_tablet_manager.cc:1428] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:23:49.263759 21311 tablet_bootstrap.cc:654] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Neither blocks nor log segments found. Creating new log.
I20260929 16:23:49.264662 21311 log.cc:824] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:23:49.266594 21311 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: No bootstrap required, opened a new log
I20260929 16:23:49.266794 21311 ts_tablet_manager.cc:1397] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:23:49.269649 21311 raft_consensus.cc:348] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.269897 21311 raft_consensus.cc:374] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:23:49.269980 21311 raft_consensus.cc:729] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ed95ea51124535b59221a78bf3230b, State: Initialized, Role: FOLLOWER
I20260929 16:23:49.270474 21311 consensus_queue.cc:260] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.271209 21297 heartbeater.cc:503] Master 127.4.200.126:45181 was elected leader, sending a full tablet report...
I20260929 16:23:49.271485 21311 ts_tablet_manager.cc:1428] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 16:23:49.437148 21298 tablet.cc:2369] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:49.465251 21314 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:23:49.465461 21314 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.466675 21314 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:49.471181 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.466953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.4.200.67:59409)
0929 16:23:49.467069 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.467085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:49.467185 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:49.467462 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:49.467465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:49.467489 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:49.467612 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.467620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.468632 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.468634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.469773 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.469784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.470692 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.470694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.470841 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.470879 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.470965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.471035 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:23:49.471237 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.467076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:59409 (local address 127.4.200.66:38027)
0929 16:23:49.467214 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.467218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.467234 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:23:49.467287 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.467289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.467333 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.467450 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:49.467747 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.468521 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.469926 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.470561 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.470707 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.470773 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.471008 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.471094 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:23:49.471515 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.467218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.4.200.67:60893)
0929 16:23:49.467646 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.467661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:23:49.467738 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:23:49.468030 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:23:49.468032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:23:49.468044 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:23:49.468133 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.468137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.468994 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.468997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.470048 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:23:49.470055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.471096 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.471099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:23:49.471189 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.471219 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:23:49.471293 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:23:49.471362 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":137,"threads_started":1}
I20260929 16:23:49.471731 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.467488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:60893 (local address 127.4.200.65:39755)
0929 16:23:49.467752 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.467756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.467770 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:23:49.467830 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.467833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.467894 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.468014 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:23:49.468247 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.468874 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.470185 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.470899 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.471051 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.471196 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.471486 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.471578 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260929 16:23:49.471891 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee834331953a4a1fbfa736f8f92b151d" candidate_uuid: "96ed95ea51124535b59221a78bf3230b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73f9c299a2742a7ac8559197319bd6c" is_pre_election: true
I20260929 16:23:49.472281 21119 raft_consensus.cc:2454] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ed95ea51124535b59221a78bf3230b in term 0.
I20260929 16:23:49.472270 20980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee834331953a4a1fbfa736f8f92b151d" candidate_uuid: "96ed95ea51124535b59221a78bf3230b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5183d72c8d84812b3d022e575c52e4b" is_pre_election: true
I20260929 16:23:49.472549 20980 raft_consensus.cc:2454] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ed95ea51124535b59221a78bf3230b in term 0.
I20260929 16:23:49.472818 21187 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c; no voters:
I20260929 16:23:49.473065 21314 raft_consensus.cc:2781] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:23:49.473173 21314 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:23:49.473229 21314 raft_consensus.cc:3037] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:49.474241 21314 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.474620 21314 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [CANDIDATE]: Term 1 election: Requested vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755)
I20260929 16:23:49.475122 20980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee834331953a4a1fbfa736f8f92b151d" candidate_uuid: "96ed95ea51124535b59221a78bf3230b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5183d72c8d84812b3d022e575c52e4b"
I20260929 16:23:49.475236 20980 raft_consensus.cc:3037] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:49.476259 20980 raft_consensus.cc:2454] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96ed95ea51124535b59221a78bf3230b in term 1.
I20260929 16:23:49.476517 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee834331953a4a1fbfa736f8f92b151d" candidate_uuid: "96ed95ea51124535b59221a78bf3230b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73f9c299a2742a7ac8559197319bd6c"
I20260929 16:23:49.476578 21187 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ed95ea51124535b59221a78bf3230b, b5183d72c8d84812b3d022e575c52e4b; no voters:
I20260929 16:23:49.476657 21119 raft_consensus.cc:3037] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:23:49.476792 21314 raft_consensus.cc:2781] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:23:49.477020 21314 raft_consensus.cc:686] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 1 LEADER]: Becoming Leader. State: Replica: 96ed95ea51124535b59221a78bf3230b, State: Running, Role: LEADER
I20260929 16:23:49.477380 21314 consensus_queue.cc:237] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:23:49.478008 21119 raft_consensus.cc:2454] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96ed95ea51124535b59221a78bf3230b in term 1.
I20260929 16:23:49.480331 20837 catalog_manager.cc:5495] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b reported cstate change: term changed from 0 to 1, leader changed from <none> to 96ed95ea51124535b59221a78bf3230b (127.4.200.67). New cstate: current_term: 1 leader_uuid: "96ed95ea51124535b59221a78bf3230b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } health_report { overall_health: UNKNOWN } } }
W20260929 16:23:49.490355 21166 tablet.cc:2369] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:23:49.543378 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.539914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.4.200.67:33325)
0929 16:23:49.540042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:23:49.540046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:23:49.540062 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:23:49.540138 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:23:49.540140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:23:49.540193 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:23:49.540290 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:23:49.540579 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.541089 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.541897 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:23:49.542115 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:23:49.542206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:23:49.542545 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:23:49.542973 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:23:49.543070 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:23:49.543177 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 16:23:49.543233 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:23:49.549679 21119 raft_consensus.cc:1261] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Refusing update from remote peer 96ed95ea51124535b59221a78bf3230b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:49.549679 20980 raft_consensus.cc:1261] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Refusing update from remote peer 96ed95ea51124535b59221a78bf3230b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:23:49.550455 21314 consensus_queue.cc:1035] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:49.550698 21317 consensus_queue.cc:1035] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:23:49.560577 21322 mvcc.cc:205] Tried to move back new op lower bound from 7334703225026863104 to 7334703224742117376. Current Snapshot: MvccSnapshot[applied={T|T < 7334703225026863104}]
I20260929 16:23:49.567800 20837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:49.568012 20837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:23:49.570933 20837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Sending DeleteTablet for 3 replicas of tablet ee834331953a4a1fbfa736f8f92b151d
I20260929 16:23:49.571925 21231 tablet_service.cc:1514] Processing DeleteTablet for tablet ee834331953a4a1fbfa736f8f92b151d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:51362
I20260929 16:23:49.572000 21099 tablet_service.cc:1514] Processing DeleteTablet for tablet ee834331953a4a1fbfa736f8f92b151d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:43600
I20260929 16:23:49.572067 20960 tablet_service.cc:1514] Processing DeleteTablet for tablet ee834331953a4a1fbfa736f8f92b151d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:23:49 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:49552
I20260929 16:23:49.572667 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20876
I20260929 16:23:49.572685 21330 tablet_replica.cc:330] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: stopping tablet replica
I20260929 16:23:49.575369 21330 raft_consensus.cc:2229] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:23:49.575821 21330 raft_consensus.cc:2258] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:49.576300 21331 tablet_replica.cc:330] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: stopping tablet replica
I20260929 16:23:49.576713 21331 raft_consensus.cc:2229] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:23:49.577126 21331 raft_consensus.cc:2258] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:23:49.577692 21330 ts_tablet_manager.cc:1905] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:23:49.578822 21331 ts_tablet_manager.cc:1905] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:23:49.580770 20825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:39755 (error 108)
I20260929 16:23:49.581198 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21030
I20260929 16:23:49.582250 21330 ts_tablet_manager.cc:1918] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:23:49.582384 21330 log.cc:1196] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/ee834331953a4a1fbfa736f8f92b151d
I20260929 16:23:49.582801 21330 ts_tablet_manager.cc:1939] T ee834331953a4a1fbfa736f8f92b151d P 96ed95ea51124535b59221a78bf3230b: Deleting consensus metadata
I20260929 16:23:49.583420 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.583046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:39755 (local address 127.0.0.1:49562)
0929 16:23:49.583245 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.583317 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 16:23:49.583699 20825 catalog_manager.cc:4584] TS b5183d72c8d84812b3d022e575c52e4b (127.4.200.65:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee834331953a4a1fbfa736f8f92b151d: Network error: Client connection negotiation failed: client connection to 127.4.200.65:39755: connect: Connection refused (error 111)
I20260929 16:23:49.584344 20823 catalog_manager.cc:4841] TS 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): tablet ee834331953a4a1fbfa736f8f92b151d (table test-table [id=c7a446002a054a67923356d0ff336f95]) successfully deleted
I20260929 16:23:49.591621 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21168
I20260929 16:23:49.592558 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:23:49.591840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.0.0.1:43602)
0929 16:23:49.592357 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:23:49.592441 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.66:38027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260929 16:23:49.597878 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 20805
I20260929 16:24:09.605055 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--webserver_interface=127.4.200.126
--webserver_port=34307
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:09.714489 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:09.714743 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:09.714819 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:09.719013 21333 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:24:09.719069 21333 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:24:09.719084 21333 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:24:09.719098 21333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:09.719117 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:09.719175 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:09.719197 21333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:09.719228 21333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:09.724617 21333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:09.725710 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:09.726742 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:09.735476 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:09.735543 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:09.735505 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:09.735598 21333 server_base.cc:1034] running on GCE node
I20260929 16:24:09.736361 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:09.737376 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:09.738548 21333 hybrid_clock.cc:648] HybridClock initialized: now 1790699049738531 us; error 29 us; skew 500 ppm
Sep 29 16:24:09 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:09.748148 21333 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:09.749214 21333 webserver.cc:460] Webserver started at http://127.4.200.126:34307/ using document root <none> and password file <none>
I20260929 16:24:09.749814 21333 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:09.749924 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:09.753968 21333 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:09.756194 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:09.757449 21333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:09.757601 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b51fa78f3bf642a7a348f59142bd6362"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "14b9d1681e4345873b2e0605e4c6cf7a"
server_key_iv: "fc85058f870bf9a371b3b387dc79f0b8"
server_key_version: "encryptionkey@0"
I20260929 16:24:09.758167 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:09.771689 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:09.774616 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:09.774817 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:09.782027 21333 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:45181
I20260929 16:24:09.782299 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:45181 every 8 connection(s)
I20260929 16:24:09.783241 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:09.787525 21401 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:24:09.789211 21401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap starting.
I20260929 16:24:09.791712 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21333
I20260929 16:24:09.792172 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:39755
--local_ip_for_outbound_sockets=127.4.200.65
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=41371
--webserver_interface=127.4.200.65
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:24:09.812467 21401 log.cc:824] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:09.826855 21401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:24:09.827332 21401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Bootstrap complete.
I20260929 16:24:09.832703 21401 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:24:09.833215 21401 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Initialized, Role: FOLLOWER
I20260929 16:24:09.833775 21401 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:24:09.833925 21401 raft_consensus.cc:388] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:09.833986 21401 raft_consensus.cc:482] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:09.834074 21401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:24:09.836732 21401 raft_consensus.cc:504] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:24:09.837103 21401 leader_election.cc:302] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b51fa78f3bf642a7a348f59142bd6362; no voters:
I20260929 16:24:09.837447 21401 leader_election.cc:288] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:24:09.837653 21405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:24:09.837927 21405 raft_consensus.cc:686] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [term 3 LEADER]: Becoming Leader. State: Replica: b51fa78f3bf642a7a348f59142bd6362, State: Running, Role: LEADER
I20260929 16:24:09.838284 21405 consensus_queue.cc:237] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } }
I20260929 16:24:09.838814 21401 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:09.839941 21407 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b51fa78f3bf642a7a348f59142bd6362. Latest consensus state: current_term: 3 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:24:09.840088 21407 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:09.839941 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b51fa78f3bf642a7a348f59142bd6362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51fa78f3bf642a7a348f59142bd6362" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 45181 } } }
I20260929 16:24:09.840423 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:09.840466 21414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:09.845628 21414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:09.846381 21414 catalog_manager.cc:1259] Loaded cluster ID: 377b59c3ca0a4bee809a927619c34562
I20260929 16:24:09.846472 21414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:09.849020 21414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:09.849769 21414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Loaded TSK: 0
I20260929 16:24:09.849856 21414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b51fa78f3bf642a7a348f59142bd6362: Loaded TSK: 1
F20260929 16:24:09.850623 21414 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2c98a359ad google::LogMessage::Fail() at ??:0
@ 0x7f2c98a3888a google::LogMessage::SendToLog() at ??:0
@ 0x7f2c98a354de google::LogMessage::Flush() at ??:0
@ 0x7f2c98a36a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2c9b6f918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2c9b6f1f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2ca339a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2ca337499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2ca3375178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f2ca33b6df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2ca33d582b std::function<>::operator()() at ??:0
@ 0x7f2ca3374b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2ca3375f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2ca3372579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2ca33b6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2c9ac873d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2c9ac87c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2c9ac893f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2c9ac78512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2c9c2bc6db start_thread at ??:0
@ 0x7f2c9753361f clone at ??:0
W20260929 16:24:09.953929 21403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:09.954301 21403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:09.954368 21403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:09.960628 21403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:09.960724 21403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:09.960835 21403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:09.969561 21403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=41371
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:09.971063 21403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:09.972421 21403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:09.983928 21451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:09.984063 21448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:09.984325 21403 server_base.cc:1034] running on GCE node
W20260929 16:24:09.984416 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:09.984805 21403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:09.985585 21403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:09.986768 21403 hybrid_clock.cc:648] HybridClock initialized: now 1790699049986738 us; error 50 us; skew 500 ppm
Sep 29 16:24:09 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:09.989768 21403 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:24:09.990829 21403 webserver.cc:460] Webserver started at http://127.4.200.65:41371/ using document root <none> and password file <none>
I20260929 16:24:09.991397 21403 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:09.991499 21403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:09.995786 21403 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:24:09.998304 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:09.999861 21403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:10.000010 21403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "b5183d72c8d84812b3d022e575c52e4b"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "3438906f4469ed8075681cdf30ffce31"
server_key_iv: "990567efaa1111a9226bac2029dc070d"
server_key_version: "encryptionkey@0"
I20260929 16:24:10.000594 21403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:10.019204 21403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:10.022358 21403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:10.022604 21403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:10.023204 21403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:10.024317 21465 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:24:10.029717 21403 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:24:10.029805 21403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:24:10.029898 21403 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:24:10.031656 21403 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:24:10.031728 21403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:10.032016 21465 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Bootstrap starting.
I20260929 16:24:10.042666 21403 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:39755
I20260929 16:24:10.042691 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:39755 every 8 connection(s)
I20260929 16:24:10.043761 21403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:10.050726 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21403
I20260929 16:24:10.051186 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:38027
--local_ip_for_outbound_sockets=127.4.200.66
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=37223
--webserver_interface=127.4.200.66
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:24:10.055119 21465 log.cc:824] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:10.064575 21465 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:24:10.065166 21465 tablet_bootstrap.cc:492] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Bootstrap complete.
I20260929 16:24:10.066742 21465 ts_tablet_manager.cc:1397] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.006s
I20260929 16:24:10.070289 21465 raft_consensus.cc:348] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:10.071003 21465 raft_consensus.cc:729] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5183d72c8d84812b3d022e575c52e4b, State: Initialized, Role: FOLLOWER
I20260929 16:24:10.071695 21465 consensus_queue.cc:260] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:10.072912 21465 ts_tablet_manager.cc:1428] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 16:24:10.119992 21575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:24:10.046142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:44369)
0929 16:24:10.046691 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.046746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:10.047613 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:10.119200 (+ 71587us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":111,"threads_started":1}
W20260929 16:24:10.120625 21573 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:45181 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:45181: Client connection negotiation failed: client connection to 127.4.200.126:45181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:24:10.121421 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.121139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:48973)
0929 16:24:10.121272 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.121347 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:24:10.122201 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.121982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:52399)
0929 16:24:10.122095 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.122150 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:24:10.122449 21573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:24:10.186378 21576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:10.186748 21576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:10.186841 21576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:10.190948 21576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:10.191036 21576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:10.191124 21576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:10.196717 21576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=37223
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:10.197854 21576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:10.198948 21576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:10.208711 21584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:10.208743 21583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:10.208799 21586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:10.209100 21576 server_base.cc:1034] running on GCE node
I20260929 16:24:10.209561 21576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:10.210250 21576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:10.211448 21576 hybrid_clock.cc:648] HybridClock initialized: now 1790699050211431 us; error 34 us; skew 500 ppm
Sep 29 16:24:10 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:10.214226 21576 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:24:10.215305 21576 webserver.cc:460] Webserver started at http://127.4.200.66:37223/ using document root <none> and password file <none>
I20260929 16:24:10.215931 21576 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:10.216024 21576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:10.219679 21576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 16:24:10.221925 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:10.223165 21576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:10.223301 21576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "e73f9c299a2742a7ac8559197319bd6c"
format_stamp: "Formatted at 2026-09-29 16:23:27 on dist-test-slave-s9ct"
server_key: "96fb41e9192824b64bdbf16c436f370c"
server_key_iv: "7a52c1727421d97784ab7c4334f12482"
server_key_version: "encryptionkey@0"
I20260929 16:24:10.223857 21576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:10.236205 21576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:10.239437 21576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:10.239735 21576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:10.240350 21576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:10.241397 21600 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 16:24:10.245141 21600 ts_tablet_manager.cc:1852] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 16:24:10.245524 21600 ts_tablet_manager.cc:1905] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:10.249346 21600 ts_tablet_manager.cc:1918] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:10.249439 21600 log.cc:1196] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:10.249840 21600 ts_tablet_manager.cc:1939] T ee834331953a4a1fbfa736f8f92b151d P e73f9c299a2742a7ac8559197319bd6c: Deleting consensus metadata
I20260929 16:24:10.250168 21576 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 16:24:10.250244 21576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260929 16:24:10.250358 21576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:10.250424 21576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:10.259687 21576 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:38027
I20260929 16:24:10.259727 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:38027 every 8 connection(s)
I20260929 16:24:10.260545 21576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:10.260874 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21576
I20260929 16:24:10.261379 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:33325
--local_ip_for_outbound_sockets=127.4.200.67
--tserver_master_addrs=127.4.200.126:45181
--webserver_port=42947
--webserver_interface=127.4.200.67
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:24:10.264380 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.263006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:36755)
0929 16:24:10.263473 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.263602 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
W20260929 16:24:10.264856 21708 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:45181 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:45181: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
I20260929 16:24:10.265496 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.265263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:35083)
0929 16:24:10.265381 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.265438 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:24:10.266167 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.265970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:45401)
0929 16:24:10.266070 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.266121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:24:10.266338 21708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:24:10.373366 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:10.373689 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:10.373768 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:10.377902 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:10.377985 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:10.378074 21710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:10.383343 21710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:33325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=42947
--tserver_master_addrs=127.4.200.126:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:10.384408 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:10.385452 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:10.395714 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:10.395613 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:10.395665 21710 server_base.cc:1034] running on GCE node
W20260929 16:24:10.395603 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:10.396385 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:10.397022 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:10.398216 21710 hybrid_clock.cc:648] HybridClock initialized: now 1790699050398171 us; error 57 us; skew 500 ppm
Sep 29 16:24:10 dist-test-slave-s9ct krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:10.401196 21710 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:24:10.402114 21710 webserver.cc:460] Webserver started at http://127.4.200.67:42947/ using document root <none> and password file <none>
I20260929 16:24:10.402769 21710 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:10.402835 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:10.407167 21710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:24:10.409289 21726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:10.410586 21710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 16:24:10.410715 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "96ed95ea51124535b59221a78bf3230b"
format_stamp: "Formatted at 2026-09-29 16:23:28 on dist-test-slave-s9ct"
server_key: "ee3154c12b2c34326253a899eea23a60"
server_key_iv: "e87dd8979942bff62fea0f619e106add"
server_key_version: "encryptionkey@0"
I20260929 16:24:10.411356 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:10.424605 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:10.427731 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:10.427963 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:10.428598 21710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:10.429651 21710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:10.429714 21710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:10.429756 21710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:10.429811 21710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:10.439065 21710 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:33325
I20260929 16:24:10.439082 21839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:33325 every 8 connection(s)
I20260929 16:24:10.440070 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:10.443866 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.442111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:33381)
0929 16:24:10.442618 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.442730 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
W20260929 16:24:10.444331 21840 heartbeater.cc:644] Failed to heartbeat to 127.4.200.126:45181 (0 consecutive failures): Network error: Failed to ping master at 127.4.200.126:45181: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
I20260929 16:24:10.444976 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.444727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:42685)
0929 16:24:10.444862 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.444919 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:24:10.445627 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:10.445405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:50449)
0929 16:24:10.445526 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:10.445580 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:24:10.445824 21840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:24:10.448307 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21710
I20260929 16:24:11.123462 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.123032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:60025)
0929 16:24:11.123269 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:11.123326 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260929 16:24:11.267282 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.266920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:55407)
0929 16:24:11.267150 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:11.267201 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260929 16:24:11.446928 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.446463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:39169)
0929 16:24:11.446766 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:11.446826 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":81,"threads_started":1}
I20260929 16:24:11.872721 21847 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:11.872936 21847 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:11.874084 21847 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
Sep 29 16:24:11 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.67@KRBTEST.COM
Sep 29 16:24:11 dist-test-slave-s9ct krb5kdc[20257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.66@KRBTEST.COM
I20260929 16:24:11.890753 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.874693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:52661 (local address 127.4.200.67:33325)
0929 16:24:11.875116 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:11.875132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:11.875184 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:24:11.876087 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:11.876097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:11.876545 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:11.876884 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:11.878666 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.880209 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.882128 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.882867 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.883039 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:11.886603 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:11.886609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:11.886621 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:11.886648 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:11.888225 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:11.888769 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:11.888775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:11.888780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:11.888898 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:11.889251 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:11.889254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:11.889256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:11.889652 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:11.889793 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:11.890299 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 16:24:11.890585 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":131,"threads_started":1}
I20260929 16:24:11.890743 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.874314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33325 (local address 127.4.200.65:52661)
0929 16:24:11.874645 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:11.874666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:11.874918 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:11.876997 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:11.877003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:11.877612 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:11.878474 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:11.878496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.880358 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.880361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:11.882009 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:11.882020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.883023 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.883028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:11.883378 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:11.884257 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:11.884284 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:11.886330 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:11.888349 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:11.888355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:11.888368 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:11.888638 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:11.888992 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:11.888997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:11.888999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:11.889148 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:11.889768 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:11.889774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:11.889992 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 16:24:11.890176 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 16:24:11.890523 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":490,"spinlock_wait_cycles":22528,"thread_start_us":93,"threads_started":1}
I20260929 16:24:11.890810 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.874654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:42713 (local address 127.4.200.66:38027)
0929 16:24:11.874981 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:11.874996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:11.875211 (+ 215us) server_negotiation.cc:407] Connection header received
0929 16:24:11.876115 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:11.876123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:11.876544 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:11.876884 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:11.878667 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.880209 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.882128 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.882867 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.883039 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:11.886802 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:11.886807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:11.886819 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:11.886847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:11.888623 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:11.889097 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:11.889101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:11.889105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:11.889210 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:11.889506 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:11.889509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:11.889511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:11.889859 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:11.889993 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:11.890333 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:24:11.890625 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260929 16:24:11.890799 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:11.874323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:38027 (local address 127.4.200.65:42713)
0929 16:24:11.875115 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:11.875125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:11.875194 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:11.876997 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:11.877003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:11.877612 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:11.878474 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:11.878496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.880360 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.880363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:11.882009 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:11.882020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:11.883027 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:11.883030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:11.883375 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:11.884257 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:11.884284 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:11.886616 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:11.888760 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:11.888763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:11.888765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:11.888974 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:11.889306 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:11.889311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:11.889313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:11.889404 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:11.889976 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:11.889980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:11.890088 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 16:24:11.890191 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:24:11.890621 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":513,"spinlock_wait_cycles":29312,"thread_start_us":78,"threads_started":1}
W20260929 16:24:11.893455 21462 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
W20260929 16:24:11.893589 21460 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:11.893702 21460 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b; no voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c
I20260929 16:24:11.893996 21847 raft_consensus.cc:2726] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:24:12.124637 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:12.124294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:51623)
0929 16:24:12.124468 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:12.124547 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:24:12.268591 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:12.268139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:58607)
0929 16:24:12.268411 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:12.268485 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260929 16:24:12.448071 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:12.447671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:43477)
0929 16:24:12.447916 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:12.447974 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260929 16:24:13.125897 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:13.125456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:52185)
0929 16:24:13.125723 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:13.125788 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260929 16:24:13.269793 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:13.269363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:37045)
0929 16:24:13.269608 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:13.269676 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260929 16:24:13.449260 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:13.448872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:34507)
0929 16:24:13.449103 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:13.449157 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260929 16:24:13.523526 21857 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:13.523660 21857 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:13.524092 21857 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
W20260929 16:24:13.524923 21460 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
W20260929 16:24:13.525009 21462 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:13.525081 21462 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b; no voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c
I20260929 16:24:13.525265 21857 raft_consensus.cc:2726] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:24:14.127513 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:14.126740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:56283)
0929 16:24:14.127306 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:14.127386 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":68,"threads_started":1}
I20260929 16:24:14.271173 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:14.270743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:47921)
0929 16:24:14.271002 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:14.271062 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260929 16:24:14.450525 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:14.450067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:54837)
0929 16:24:14.450302 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:14.450358 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260929 16:24:15.128794 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:15.128372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:50049)
0929 16:24:15.128646 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:15.128701 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260929 16:24:15.241835 21862 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:15.242012 21862 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:15.242501 21862 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
W20260929 16:24:15.243299 21462 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
W20260929 16:24:15.243386 21460 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:15.243489 21460 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b; no voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c
I20260929 16:24:15.243631 21862 raft_consensus.cc:2726] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:24:15.272401 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:15.271948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:54529)
0929 16:24:15.272248 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:15.272309 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260929 16:24:15.451865 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:15.451403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:55907)
0929 16:24:15.451698 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:15.451769 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260929 16:24:16.130029 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:16.129580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:41627)
0929 16:24:16.129862 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:16.129923 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260929 16:24:16.273928 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:16.273451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:44559)
0929 16:24:16.273762 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:16.273829 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260929 16:24:16.453125 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:16.452736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:39819)
0929 16:24:16.452974 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:16.453030 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260929 16:24:17.131311 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:17.130860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:47519)
0929 16:24:17.131145 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:17.131209 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260929 16:24:17.275255 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:17.274809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:45077)
0929 16:24:17.275095 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:17.275157 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":74,"threads_started":1}
I20260929 16:24:17.454353 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:17.453893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:35095)
0929 16:24:17.454199 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:17.454260 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260929 16:24:18.132570 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:18.132180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:55519)
0929 16:24:18.132416 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:18.132470 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260929 16:24:18.276438 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:18.276037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:45335)
0929 16:24:18.276286 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:18.276345 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260929 16:24:18.455628 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:18.455229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:34015)
0929 16:24:18.455484 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:18.455537 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260929 16:24:19.133766 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:19.133349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:50245)
0929 16:24:19.133607 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:19.133668 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260929 16:24:19.277945 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:19.277471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:36663)
0929 16:24:19.277784 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:19.277847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260929 16:24:19.456900 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:19.456425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:57275)
0929 16:24:19.456733 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:19.456796 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260929 16:24:20.135140 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:20.134679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:57993)
0929 16:24:20.134923 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:20.134995 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260929 16:24:20.279376 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:20.278912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:42357)
0929 16:24:20.279185 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:20.279269 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
I20260929 16:24:20.286303 21879 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:20.286501 21879 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:20.286891 21879 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
W20260929 16:24:20.287621 21462 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
W20260929 16:24:20.287714 21460 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:20.287823 21460 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b; no voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c
I20260929 16:24:20.287983 21879 raft_consensus.cc:2726] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:24:20.458057 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:20.457584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:60467)
0929 16:24:20.457876 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:20.457953 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260929 16:24:21.136325 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:21.135900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:48725)
0929 16:24:21.136157 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:21.136223 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260929 16:24:21.280781 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:21.280335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:51679)
0929 16:24:21.280620 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:21.280683 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260929 16:24:21.459316 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:21.458885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:59453)
0929 16:24:21.459154 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:21.459218 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260929 16:24:22.137521 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:22.137073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:42619)
0929 16:24:22.137362 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:22.137424 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260929 16:24:22.282101 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:22.281629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:52205)
0929 16:24:22.281920 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:22.282000 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260929 16:24:22.460551 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:22.460117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:59357)
0929 16:24:22.460391 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:22.460454 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260929 16:24:23.139124 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:23.138380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:32923)
0929 16:24:23.138827 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:23.138937 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":119,"threads_started":1}
I20260929 16:24:23.283399 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:23.282905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:49187)
0929 16:24:23.283220 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:23.283302 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260929 16:24:23.461753 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:23.461361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:33931)
0929 16:24:23.461592 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:23.461662 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260929 16:24:23.769423 21890 raft_consensus.cc:482] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:23.769598 21890 raft_consensus.cc:504] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73f9c299a2742a7ac8559197319bd6c" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 38027 } } peers { permanent_uuid: "96ed95ea51124535b59221a78bf3230b" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33325 } } peers { permanent_uuid: "b5183d72c8d84812b3d022e575c52e4b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 39755 } }
I20260929 16:24:23.770102 21890 leader_election.cc:288] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027), 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325)
W20260929 16:24:23.770843 21462 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e73f9c299a2742a7ac8559197319bd6c (127.4.200.66:38027): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
W20260929 16:24:23.770947 21460 leader_election.cc:341] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96ed95ea51124535b59221a78bf3230b (127.4.200.67:33325): Not found: Tablet not found: ee834331953a4a1fbfa736f8f92b151d
I20260929 16:24:23.771060 21460 leader_election.cc:302] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5183d72c8d84812b3d022e575c52e4b; no voters: 96ed95ea51124535b59221a78bf3230b, e73f9c299a2742a7ac8559197319bd6c
I20260929 16:24:23.771217 21890 raft_consensus.cc:2726] T ee834331953a4a1fbfa736f8f92b151d P b5183d72c8d84812b3d022e575c52e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:24:24.140386 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:24.139978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:45417)
0929 16:24:24.140230 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:24.140287 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260929 16:24:24.284675 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:24.284180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:43043)
0929 16:24:24.284516 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:24.284580 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":76,"threads_started":1}
I20260929 16:24:24.463057 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:24.462607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:56305)
0929 16:24:24.462873 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:24.462950 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260929 16:24:25.141846 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:25.141406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.65:48619)
0929 16:24:25.141677 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:25.141738 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260929 16:24:25.286059 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:25.285632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.66:38751)
0929 16:24:25.285904 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:25.285963 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
I20260929 16:24:25.453455 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21403
I20260929 16:24:25.459280 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21576
I20260929 16:24:25.464375 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:25.463982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:45181 (local address 127.4.200.67:59215)
0929 16:24:25.464237 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:25.464289 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.126:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260929 16:24:25.464766 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21710
2026-09-29T16:24:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59439 ms)
[----------] 30 tests from SecurityITest (244002 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:25 dist-test-slave-s9ct krb5kdc[21899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:25 dist-test-slave-s9ct krb5kdc[21899](info): set up 2 sockets
Sep 29 16:24:25 dist-test-slave-s9ct krb5kdc[21899](info): commencing operation
krb5kdc: starting...
W20260929 16:24:26.522250 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:26 dist-test-slave-s9ct krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:26.649797 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:38005
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:26.759099 21912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:26.759414 21912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:26.759496 21912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:26.763484 21912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:26.763578 21912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:26.763620 21912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:26.763676 21912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:26.763711 21912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:26.768930 21912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:38005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:26.770005 21912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:26.771054 21912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:26.779932 21918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:26.779868 21917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:26.779896 21920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:26.780098 21912 server_base.cc:1034] running on GCE node
I20260929 16:24:26.780735 21912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:26.781806 21912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:26.782989 21912 hybrid_clock.cc:648] HybridClock initialized: now 1790699066782964 us; error 41 us; skew 500 ppm
Sep 29 16:24:26 dist-test-slave-s9ct krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699066, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:26.785660 21912 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:26.786626 21912 webserver.cc:460] Webserver started at http://127.4.200.126:36291/ using document root <none> and password file <none>
I20260929 16:24:26.787209 21912 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:26.787300 21912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:26.787531 21912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:26.789530 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "3f640f39768e464ca70b17dc64ba0989"
format_stamp: "Formatted at 2026-09-29 16:24:26 on dist-test-slave-s9ct"
server_key: "d4d08dc41b28d0cedf274f756259eccb"
server_key_iv: "74102a4059d5ee5d8d01348ae36ce6a3"
server_key_version: "encryptionkey@0"
I20260929 16:24:26.790097 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "3f640f39768e464ca70b17dc64ba0989"
format_stamp: "Formatted at 2026-09-29 16:24:26 on dist-test-slave-s9ct"
server_key: "d4d08dc41b28d0cedf274f756259eccb"
server_key_iv: "74102a4059d5ee5d8d01348ae36ce6a3"
server_key_version: "encryptionkey@0"
I20260929 16:24:26.793807 21912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:24:26.796367 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:26.797698 21912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:26.797847 21912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "3f640f39768e464ca70b17dc64ba0989"
format_stamp: "Formatted at 2026-09-29 16:24:26 on dist-test-slave-s9ct"
server_key: "d4d08dc41b28d0cedf274f756259eccb"
server_key_iv: "74102a4059d5ee5d8d01348ae36ce6a3"
server_key_version: "encryptionkey@0"
I20260929 16:24:26.797972 21912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:26.816301 21912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:26.819447 21912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:26.819674 21912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:26.827270 21912 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:38005
I20260929 16:24:26.827280 21979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:38005 every 8 connection(s)
I20260929 16:24:26.828246 21912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:26.831096 21980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:26.836278 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 21912
I20260929 16:24:26.836404 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:26.836730 4897 external_mini_cluster.cc:1428] Setting key fefaa7ee3102fae4f50d655f4873c6e1
I20260929 16:24:26.836889 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: Bootstrap starting.
I20260929 16:24:26.839521 21980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:26.840452 21980 log.cc:824] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:26.842608 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: No bootstrap required, opened a new log
Sep 29 16:24:26 dist-test-slave-s9ct krb5kdc[21899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:26.847257 21980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } }
I20260929 16:24:26.847561 21980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:26.847644 21980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f640f39768e464ca70b17dc64ba0989, State: Initialized, Role: FOLLOWER
I20260929 16:24:26.848168 21980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } }
I20260929 16:24:26.848371 21980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:26.848479 21980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:26.848593 21980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:26.849541 21980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } }
I20260929 16:24:26.849903 21980 leader_election.cc:302] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f640f39768e464ca70b17dc64ba0989; no voters:
I20260929 16:24:26.850270 21980 leader_election.cc:288] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:26.850365 21985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:26.850561 21985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [term 1 LEADER]: Becoming Leader. State: Replica: 3f640f39768e464ca70b17dc64ba0989, State: Running, Role: LEADER
I20260929 16:24:26.850932 21985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } }
I20260929 16:24:26.851656 21980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:26.852227 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f640f39768e464ca70b17dc64ba0989. Latest consensus state: current_term: 1 leader_uuid: "3f640f39768e464ca70b17dc64ba0989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } } }
I20260929 16:24:26.852358 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:26.852798 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f640f39768e464ca70b17dc64ba0989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f640f39768e464ca70b17dc64ba0989" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 38005 } } }
I20260929 16:24:26.852906 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:26.852969 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:26.838135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.4.200.126:38005)
0929 16:24:26.838602 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:26.838617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:26.838652 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:24:26.839330 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:26.839347 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:26.839734 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:26.840047 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:26.840979 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:26.842230 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:26.843418 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:26.843920 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:26.844043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:26.847290 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:26.847296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:26.847312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:26.847344 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:26.849266 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:26.849783 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:26.849796 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:26.849802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:26.849919 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:26.850228 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:26.850231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:26.850233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:26.850844 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:26.851064 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:26.851373 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:24:26.851860 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":150,"threads_started":1}
W20260929 16:24:26.856529 22000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:26.856631 22000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:26.856731 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:26.857473 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:26.863435 21990 catalog_manager.cc:1347] Generated new cluster ID: 680f7808fac249c783b88328180efa32
I20260929 16:24:26.863508 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:26.878008 21990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:26.879459 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:26.892517 21990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: Generated new TSK 0
I20260929 16:24:26.893360 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:24:26.930979 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:27.046923 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:27.047211 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:27.047257 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:27.051925 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:27.052006 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:27.052106 22006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:27.057860 22006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:27.058990 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:27.059985 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:27.069882 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.069880 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.069882 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:27.070259 22006 server_base.cc:1034] running on GCE node
I20260929 16:24:27.070765 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:27.071391 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:27.072609 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790699067072571 us; error 56 us; skew 500 ppm
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:27.075600 22006 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:24:27.076566 22006 webserver.cc:460] Webserver started at http://127.4.200.65:42417/ using document root <none> and password file <none>
I20260929 16:24:27.077194 22006 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:27.077261 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:27.077445 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:27.079559 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "af34e12b03db46358225fe43928358d6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "458c7416dd9c9d00c23c1892b7ccd900"
server_key_iv: "956464cf4d71b0e9c888803a2e2c7e04"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.080096 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "af34e12b03db46358225fe43928358d6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "458c7416dd9c9d00c23c1892b7ccd900"
server_key_iv: "956464cf4d71b0e9c888803a2e2c7e04"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.083884 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:27.086465 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.087803 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:27.087934 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "af34e12b03db46358225fe43928358d6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "458c7416dd9c9d00c23c1892b7ccd900"
server_key_iv: "956464cf4d71b0e9c888803a2e2c7e04"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.088060 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:27.100543 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:27.103974 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:27.104197 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:27.104836 22006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:27.105749 22006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:27.105822 22006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.105888 22006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:27.105934 22006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.116108 22006 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:34829
I20260929 16:24:27.116169 22134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:34829 every 8 connection(s)
I20260929 16:24:27.117126 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:27.117542 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22006
I20260929 16:24:27.117702 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:27.118070 4897 external_mini_cluster.cc:1428] Setting key 6fa65e3cf7b6b72ae81632b89de6f32a
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:24:27.133755 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.119410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:54401 (local address 127.4.200.126:38005)
0929 16:24:27.119570 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.119574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.120984 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 16:24:27.122152 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.122166 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.122242 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.122343 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:27.123647 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.124568 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.125831 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.126241 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.126441 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.129006 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:27.129010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:27.129012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:27.129039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:27.130925 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.131771 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.131774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.131776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.131865 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.132349 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.132352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.132353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.132563 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:27.132677 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.133376 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.133539 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:24:27.134572 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.119963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38005 (local address 127.4.200.65:54401)
0929 16:24:27.120810 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.120871 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.121938 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.122579 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.122606 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.122985 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:27.123479 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.123493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.124722 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.124725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.125701 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.125711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.126477 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.126482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.126663 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.127315 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:27.127332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:27.128826 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:27.131097 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.131102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.131117 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.131649 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.131970 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.131976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.131981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.132206 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.132705 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:27.132715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:27.133102 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.133275 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.133552 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":118,"threads_started":1}
I20260929 16:24:27.136076 22135 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38005
I20260929 16:24:27.136349 22135 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:27.137039 22135 heartbeater.cc:511] Master 127.4.200.126:38005 requested a full tablet report, sending...
I20260929 16:24:27.138844 21944 ts_manager.cc:194] Registered new tserver with Master: af34e12b03db46358225fe43928358d6 (127.4.200.65:34829)
I20260929 16:24:27.140007 21944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:54401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:24:27.150712 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:27.267431 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:27.267724 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:27.267802 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:27.272027 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:27.272130 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:27.272233 22140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:27.277678 22140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:27.278891 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:27.279945 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:27.290136 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.290158 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:27.290244 22140 server_base.cc:1034] running on GCE node
W20260929 16:24:27.290217 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:27.290964 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:27.291718 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:27.292937 22140 hybrid_clock.cc:648] HybridClock initialized: now 1790699067292920 us; error 31 us; skew 500 ppm
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:27.296263 22140 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:24:27.297690 22140 webserver.cc:460] Webserver started at http://127.4.200.66:39377/ using document root <none> and password file <none>
I20260929 16:24:27.298612 22140 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:27.298727 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:27.298936 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:27.301277 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "5ac1db83115140b0926c2abdf861a9ef"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "87f80e6f3f8aba0566725ebc5e9de138"
server_key_iv: "c63b2c9cf192dfd9acdf6cb41c9942b9"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.301838 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "5ac1db83115140b0926c2abdf861a9ef"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "87f80e6f3f8aba0566725ebc5e9de138"
server_key_iv: "c63b2c9cf192dfd9acdf6cb41c9942b9"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.305850 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:27.308568 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.310051 22140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:27.310171 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "5ac1db83115140b0926c2abdf861a9ef"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "87f80e6f3f8aba0566725ebc5e9de138"
server_key_iv: "c63b2c9cf192dfd9acdf6cb41c9942b9"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.310272 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:27.327083 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:27.331485 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:27.331694 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:27.332312 22140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:27.333204 22140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:27.333256 22140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.333299 22140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:27.333315 22140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.343168 22140 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:45499
I20260929 16:24:27.343184 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:45499 every 8 connection(s)
I20260929 16:24:27.344149 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:27.348757 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22140
I20260929 16:24:27.348908 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:27.349231 4897 external_mini_cluster.cc:1428] Setting key add2244515a0902f4c58749674b7cb12
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:24:27.360705 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.346155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:52967 (local address 127.4.200.126:38005)
0929 16:24:27.346302 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.346306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.347523 (+ 1217us) server_negotiation.cc:407] Connection header received
0929 16:24:27.348653 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.348657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.348723 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.348832 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:27.350375 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.351096 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.353052 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.353385 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.353452 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.356668 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:27.356675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:27.356683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:27.356720 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:27.358431 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.358955 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.358959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.358960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.359062 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.359406 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.359411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.359413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.359684 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:27.359802 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.360325 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.360480 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:24:27.361518 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.346604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38005 (local address 127.4.200.66:52967)
0929 16:24:27.347334 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.347376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.348330 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.349119 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.349127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.349574 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:27.350218 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.350235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.351259 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.351266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.352900 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.352913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.353688 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.353693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.353913 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.354661 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:27.354688 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:27.356468 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:27.358546 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.358551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.358564 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.358844 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.359169 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.359171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.359172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.359309 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.359804 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:27.359809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:27.360055 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.360234 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.360573 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":129,"threads_started":1}
I20260929 16:24:27.362848 22269 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38005
I20260929 16:24:27.363140 22269 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:24:27.363683 22269 heartbeater.cc:511] Master 127.4.200.126:38005 requested a full tablet report, sending...
I20260929 16:24:27.364753 21944 ts_manager.cc:194] Registered new tserver with Master: 5ac1db83115140b0926c2abdf861a9ef (127.4.200.66:45499)
I20260929 16:24:27.365623 21944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:52967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:24:27.383261 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:27.493336 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:27.493606 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:27.493650 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:27.497828 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:27.497928 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:27.497993 22274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:27.503437 22274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:27.504398 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:27.505352 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:27.514985 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.515188 22282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.515198 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:27.515429 22274 server_base.cc:1034] running on GCE node
I20260929 16:24:27.515908 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:27.516674 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:27.517894 22274 hybrid_clock.cc:648] HybridClock initialized: now 1790699067517869 us; error 40 us; skew 500 ppm
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:27.521124 22274 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:24:27.522094 22274 webserver.cc:460] Webserver started at http://127.4.200.67:36687/ using document root <none> and password file <none>
I20260929 16:24:27.522791 22274 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:27.522893 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:27.523155 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:27.525379 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "e435d963c0b4427b927eaef51b952dc6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "84149027eb98bcfe6b4d2ecc41401888"
server_key_iv: "4270676cb72ca6da6fcec2f77278932b"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.525993 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "e435d963c0b4427b927eaef51b952dc6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "84149027eb98bcfe6b4d2ecc41401888"
server_key_iv: "4270676cb72ca6da6fcec2f77278932b"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.530270 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260929 16:24:27.533011 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.534366 22274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:27.534600 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "e435d963c0b4427b927eaef51b952dc6"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "84149027eb98bcfe6b4d2ecc41401888"
server_key_iv: "4270676cb72ca6da6fcec2f77278932b"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.534727 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:27.558136 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:27.561780 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:27.562005 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:27.562716 22274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:27.563584 22274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:27.563663 22274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.563737 22274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:27.563792 22274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.573307 22274 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:42211
I20260929 16:24:27.573334 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:42211 every 8 connection(s)
I20260929 16:24:27.574236 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:27.579893 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22274
I20260929 16:24:27.580068 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:27.580363 4897 external_mini_cluster.cc:1428] Setting key ae3eba0dc1b296d4416704e66b6a32a2
Sep 29 16:24:27 dist-test-slave-s9ct krb5kdc[21899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:27.587561 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.576079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:50623 (local address 127.4.200.126:38005)
0929 16:24:27.576216 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.576219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.577298 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 16:24:27.578180 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.578183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.578242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.578349 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:27.579520 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.580135 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.581522 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.581791 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.581894 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.584315 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:27.584318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:27.584320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:27.584344 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:27.585603 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.586093 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.586095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.586097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.586175 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.586527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.586529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.586530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.586726 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:27.586806 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.587339 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.587438 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:24:27.588409 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.576439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:38005 (local address 127.4.200.67:50623)
0929 16:24:27.577147 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.577194 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.577969 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.578550 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.578557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.578905 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:27.579376 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.579389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.580285 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.580295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.581404 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.581413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.581933 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.581936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.582090 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.582746 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:27.582773 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:27.584164 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:27.585720 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.585724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.585736 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.585986 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.586253 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:27.586255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:27.586256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:27.586390 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:27.586878 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:27.586884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:27.587117 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.587276 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.587536 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":120,"threads_started":1}
I20260929 16:24:27.589489 22403 heartbeater.cc:347] Connected to a master server at 127.4.200.126:38005
I20260929 16:24:27.589736 22403 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:27.590238 22403 heartbeater.cc:511] Master 127.4.200.126:38005 requested a full tablet report, sending...
I20260929 16:24:27.591377 21944 ts_manager.cc:194] Registered new tserver with Master: e435d963c0b4427b927eaef51b952dc6 (127.4.200.67:42211)
I20260929 16:24:27.591953 21944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:50623
I20260929 16:24:27.594275 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:27.604296 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.596317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.4.200.126:38005)
0929 16:24:27.596462 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.596467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.596605 (+ 138us) server_negotiation.cc:407] Connection header received
0929 16:24:27.596771 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.596773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.596813 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.596897 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:27.597781 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.598540 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.599416 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.599641 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.599782 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.601113 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:27.601119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:27.601127 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:27.601151 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:27.602568 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.603019 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.603024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.603025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.603107 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:27.603389 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:27.603393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:27.603394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:27.603614 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:27.603761 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.604003 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.604128 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:24:27.607213 21944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:27.610112 21944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:27.625319 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.619335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.4.200.66:45499)
0929 16:24:27.619759 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.619777 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.620551 (+ 774us) server_negotiation.cc:407] Connection header received
0929 16:24:27.620650 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.620658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.620880 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.621024 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:27.621576 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.622335 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.623798 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.624612 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.624734 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.624889 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.625053 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.625145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":153,"threads_started":1}
I20260929 16:24:27.625313 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.619208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:45499 (local address 127.0.0.1:56276)
0929 16:24:27.620216 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.620269 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.620408 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.621225 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.621232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.621255 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:27.621459 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.621466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.622510 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.622514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.623661 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.623671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.624734 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.624738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.624854 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.624891 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.625007 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.625145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":56,"spinlock_wait_cycles":9600,"thread_start_us":101,"threads_started":1}
I20260929 16:24:27.626196 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.619861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:34829 (local address 127.0.0.1:55380)
0929 16:24:27.620216 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.620269 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.620408 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.621339 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.621342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.621359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:27.621481 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.621489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.623176 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.623180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.624299 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.624309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.625783 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.625786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.625884 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.625908 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.625986 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.626064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":6,"spinlock_wait_cycles":2304,"thread_start_us":108,"threads_started":1}
I20260929 16:24:27.626518 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.619208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:42211 (local address 127.0.0.1:52042)
0929 16:24:27.620647 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.620659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.620742 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.621247 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.621254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.621279 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:27.621481 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.621489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.623327 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.623331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.624427 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.624438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.625982 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.625988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.626089 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.626113 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.626189 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.626375 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":713,"thread_start_us":130,"threads_started":1}
I20260929 16:24:27.627058 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.619472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.4.200.67:42211)
0929 16:24:27.620602 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.620615 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.620828 (+ 213us) server_negotiation.cc:407] Connection header received
0929 16:24:27.620908 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.620913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.621095 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.621244 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:27.621640 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.622997 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.624596 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.625646 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.625757 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.625906 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.626796 (+ 890us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.626904 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":129,"threads_started":1}
I20260929 16:24:27.627074 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.620048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.4.200.65:34829)
0929 16:24:27.620813 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.620831 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.620854 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:24:27.620939 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.620945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.621156 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.621641 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:27.621806 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.623015 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.624439 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.625365 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.625762 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.625906 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.626030 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.626140 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":103,"threads_started":1}
I20260929 16:24:27.628563 22203 tablet_service.cc:1467] Processing CreateTablet for tablet f2034934a0fa4999bf3faf9127c44928 (DEFAULT_TABLE table=test-table [id=14249467f6a44345a540d25eea2896fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:27.629539 22203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2034934a0fa4999bf3faf9127c44928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:27.629629 22069 tablet_service.cc:1467] Processing CreateTablet for tablet f2034934a0fa4999bf3faf9127c44928 (DEFAULT_TABLE table=test-table [id=14249467f6a44345a540d25eea2896fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:27.630610 22069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2034934a0fa4999bf3faf9127c44928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:27.636482 22421 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: Bootstrap starting.
I20260929 16:24:27.636505 22337 tablet_service.cc:1467] Processing CreateTablet for tablet f2034934a0fa4999bf3faf9127c44928 (DEFAULT_TABLE table=test-table [id=14249467f6a44345a540d25eea2896fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:27.637475 22337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2034934a0fa4999bf3faf9127c44928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:27.637504 22420 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Bootstrap starting.
I20260929 16:24:27.638945 22421 tablet_bootstrap.cc:654] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:27.639732 22420 tablet_bootstrap.cc:654] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:27.639851 22421 log.cc:824] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:27.641038 22420 log.cc:824] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:27.642678 22421 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: No bootstrap required, opened a new log
I20260929 16:24:27.642890 22421 ts_tablet_manager.cc:1397] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:24:27.643056 22420 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: No bootstrap required, opened a new log
I20260929 16:24:27.643250 22420 ts_tablet_manager.cc:1397] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:27.644143 22424 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Bootstrap starting.
I20260929 16:24:27.645763 22424 tablet_bootstrap.cc:654] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:27.646806 22424 log.cc:824] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:27.647524 22420 raft_consensus.cc:348] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.647662 22421 raft_consensus.cc:348] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.647838 22420 raft_consensus.cc:374] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:27.647890 22421 raft_consensus.cc:374] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:27.647898 22420 raft_consensus.cc:729] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac1db83115140b0926c2abdf861a9ef, State: Initialized, Role: FOLLOWER
I20260929 16:24:27.647941 22421 raft_consensus.cc:729] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af34e12b03db46358225fe43928358d6, State: Initialized, Role: FOLLOWER
I20260929 16:24:27.648403 22420 consensus_queue.cc:260] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.648403 22421 consensus_queue.cc:260] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.648617 22424 tablet_bootstrap.cc:492] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: No bootstrap required, opened a new log
I20260929 16:24:27.648859 22424 ts_tablet_manager.cc:1397] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:24:27.649480 22421 ts_tablet_manager.cc:1428] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260929 16:24:27.649480 22420 ts_tablet_manager.cc:1428] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:24:27.649868 22135 heartbeater.cc:503] Master 127.4.200.126:38005 was elected leader, sending a full tablet report...
I20260929 16:24:27.650068 22269 heartbeater.cc:503] Master 127.4.200.126:38005 was elected leader, sending a full tablet report...
I20260929 16:24:27.652580 22424 raft_consensus.cc:348] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.652822 22424 raft_consensus.cc:374] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:27.652881 22424 raft_consensus.cc:729] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e435d963c0b4427b927eaef51b952dc6, State: Initialized, Role: FOLLOWER
I20260929 16:24:27.653386 22424 consensus_queue.cc:260] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.654003 22403 heartbeater.cc:503] Master 127.4.200.126:38005 was elected leader, sending a full tablet report...
I20260929 16:24:27.654386 22424 ts_tablet_manager.cc:1428] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:24:27.709697 22428 raft_consensus.cc:482] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:27.709949 22428 raft_consensus.cc:504] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.711282 22428 leader_election.cc:288] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ac1db83115140b0926c2abdf861a9ef (127.4.200.66:45499), af34e12b03db46358225fe43928358d6 (127.4.200.65:34829)
I20260929 16:24:27.715219 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.711761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:43837 (local address 127.4.200.66:45499)
0929 16:24:27.711898 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.711903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.711918 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:24:27.712068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.712072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.712122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.712231 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:27.712522 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.713024 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.714133 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.714648 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.714777 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.714831 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.715025 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.715088 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:24:27.715219 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.711666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:45499 (local address 127.4.200.67:43837)
0929 16:24:27.711811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.711830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.711940 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.712283 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.712286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.712304 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:27.712415 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.712419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.713135 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.713139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.714017 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.714025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.714773 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.714776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.714881 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.714904 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.715000 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.715078 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":20}
I20260929 16:24:27.715728 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.711771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:45053 (local address 127.4.200.65:34829)
0929 16:24:27.711914 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.711919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.712234 (+ 315us) server_negotiation.cc:407] Connection header received
0929 16:24:27.712374 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.712377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.712427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.712548 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:27.712801 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.713297 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.714424 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.714968 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.715106 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.715200 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.715527 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.715608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:24:27.716076 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.711666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:34829 (local address 127.4.200.67:45053)
0929 16:24:27.712159 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.712170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:27.712258 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:27.712562 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:27.712565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:27.712581 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:27.712697 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.712702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.713408 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.713414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.714275 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:27.714284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.715278 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.715282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:27.715365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.715394 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:24:27.715878 (+ 484us) client_negotiation.cc:240] Negotiation successful
0929 16:24:27.715949 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":154,"thread_start_us":147,"threads_started":1}
I20260929 16:24:27.716101 22223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2034934a0fa4999bf3faf9127c44928" candidate_uuid: "e435d963c0b4427b927eaef51b952dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac1db83115140b0926c2abdf861a9ef" is_pre_election: true
I20260929 16:24:27.716472 22223 raft_consensus.cc:2454] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e435d963c0b4427b927eaef51b952dc6 in term 0.
I20260929 16:24:27.716706 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2034934a0fa4999bf3faf9127c44928" candidate_uuid: "e435d963c0b4427b927eaef51b952dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af34e12b03db46358225fe43928358d6" is_pre_election: true
I20260929 16:24:27.717000 22089 raft_consensus.cc:2454] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e435d963c0b4427b927eaef51b952dc6 in term 0.
I20260929 16:24:27.716975 22292 leader_election.cc:302] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac1db83115140b0926c2abdf861a9ef, e435d963c0b4427b927eaef51b952dc6; no voters:
I20260929 16:24:27.717250 22428 raft_consensus.cc:2781] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:27.717334 22428 raft_consensus.cc:482] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:27.717383 22428 raft_consensus.cc:3037] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:27.718386 22428 raft_consensus.cc:504] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.718798 22428 leader_election.cc:288] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [CANDIDATE]: Term 1 election: Requested vote from peers 5ac1db83115140b0926c2abdf861a9ef (127.4.200.66:45499), af34e12b03db46358225fe43928358d6 (127.4.200.65:34829)
I20260929 16:24:27.719151 22223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2034934a0fa4999bf3faf9127c44928" candidate_uuid: "e435d963c0b4427b927eaef51b952dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac1db83115140b0926c2abdf861a9ef"
I20260929 16:24:27.719242 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2034934a0fa4999bf3faf9127c44928" candidate_uuid: "e435d963c0b4427b927eaef51b952dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af34e12b03db46358225fe43928358d6"
I20260929 16:24:27.719296 22223 raft_consensus.cc:3037] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:27.719386 22089 raft_consensus.cc:3037] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:27.720350 22223 raft_consensus.cc:2454] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e435d963c0b4427b927eaef51b952dc6 in term 1.
I20260929 16:24:27.720503 22089 raft_consensus.cc:2454] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e435d963c0b4427b927eaef51b952dc6 in term 1.
I20260929 16:24:27.720670 22292 leader_election.cc:302] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac1db83115140b0926c2abdf861a9ef, e435d963c0b4427b927eaef51b952dc6; no voters:
I20260929 16:24:27.720865 22428 raft_consensus.cc:2781] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:27.721113 22428 raft_consensus.cc:686] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 1 LEADER]: Becoming Leader. State: Replica: e435d963c0b4427b927eaef51b952dc6, State: Running, Role: LEADER
I20260929 16:24:27.721431 22428 consensus_queue.cc:237] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } }
I20260929 16:24:27.724609 21944 catalog_manager.cc:5495] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e435d963c0b4427b927eaef51b952dc6 (127.4.200.67). New cstate: current_term: 1 leader_uuid: "e435d963c0b4427b927eaef51b952dc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e435d963c0b4427b927eaef51b952dc6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 42211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 } health_report { overall_health: UNKNOWN } } }
I20260929 16:24:27.747773 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.744349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.4.200.67:42211)
0929 16:24:27.744484 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:27.744487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:27.744502 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:24:27.744630 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:27.744633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:27.744686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:27.744777 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:27.745052 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.745569 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.746465 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:27.746675 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:27.746752 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:27.747005 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:27.747411 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:27.747499 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:27.747550 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:24:27.747596 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:24:27.753945 22089 raft_consensus.cc:1261] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 1 FOLLOWER]: Refusing update from remote peer e435d963c0b4427b927eaef51b952dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:27.753945 22223 raft_consensus.cc:1261] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 1 FOLLOWER]: Refusing update from remote peer e435d963c0b4427b927eaef51b952dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:27.754740 22428 consensus_queue.cc:1035] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af34e12b03db46358225fe43928358d6" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:27.754999 22430 consensus_queue.cc:1035] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ac1db83115140b0926c2abdf861a9ef" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:27.763175 22435 mvcc.cc:205] Tried to move back new op lower bound from 7334703381512278016 to 7334703381389463552. Current Snapshot: MvccSnapshot[applied={T|T < 7334703381512278016}]
I20260929 16:24:27.764496 22433 mvcc.cc:205] Tried to move back new op lower bound from 7334703381512278016 to 7334703381389463552. Current Snapshot: MvccSnapshot[applied={T|T < 7334703381512278016}]
I20260929 16:24:27.773162 21943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:27.773334 21943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:27.776062 21943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f640f39768e464ca70b17dc64ba0989: Sending DeleteTablet for 3 replicas of tablet f2034934a0fa4999bf3faf9127c44928
I20260929 16:24:27.776949 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet f2034934a0fa4999bf3faf9127c44928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:27 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:52042
I20260929 16:24:27.776949 22069 tablet_service.cc:1514] Processing DeleteTablet for tablet f2034934a0fa4999bf3faf9127c44928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:27 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:55380
I20260929 16:24:27.776943 22203 tablet_service.cc:1514] Processing DeleteTablet for tablet f2034934a0fa4999bf3faf9127c44928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:27 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:56276
I20260929 16:24:27.777508 22445 tablet_replica.cc:330] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: stopping tablet replica
I20260929 16:24:27.777719 22444 tablet_replica.cc:330] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6: stopping tablet replica
I20260929 16:24:27.777974 22445 raft_consensus.cc:2229] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:27.778059 22444 raft_consensus.cc:2229] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:27.778448 22445 raft_consensus.cc:2258] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:27.778468 22444 raft_consensus.cc:2258] T f2034934a0fa4999bf3faf9127c44928 P af34e12b03db46358225fe43928358d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:27.778534 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22006
I20260929 16:24:27.779986 22443 tablet_replica.cc:330] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: stopping tablet replica
I20260929 16:24:27.780341 22443 raft_consensus.cc:2229] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:24:27.780751 22445 ts_tablet_manager.cc:1905] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:27.780825 22443 raft_consensus.cc:2258] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:27.783720 22445 ts_tablet_manager.cc:1918] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:27.783898 22445 log.cc:1196] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/f2034934a0fa4999bf3faf9127c44928
I20260929 16:24:27.783895 22443 ts_tablet_manager.cc:1905] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:27.784360 22445 ts_tablet_manager.cc:1939] T f2034934a0fa4999bf3faf9127c44928 P 5ac1db83115140b0926c2abdf861a9ef: Deleting consensus metadata
I20260929 16:24:27.785717 21930 catalog_manager.cc:4841] TS 5ac1db83115140b0926c2abdf861a9ef (127.4.200.66:45499): tablet f2034934a0fa4999bf3faf9127c44928 (table test-table [id=14249467f6a44345a540d25eea2896fa]) successfully deleted
W20260929 16:24:27.786891 21929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:34829 (error 108)
I20260929 16:24:27.787135 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22140
I20260929 16:24:27.788115 22443 ts_tablet_manager.cc:1918] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:27.788228 22443 log.cc:1196] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/f2034934a0fa4999bf3faf9127c44928
I20260929 16:24:27.788610 22443 ts_tablet_manager.cc:1939] T f2034934a0fa4999bf3faf9127c44928 P e435d963c0b4427b927eaef51b952dc6: Deleting consensus metadata
I20260929 16:24:27.789975 21931 catalog_manager.cc:4841] TS e435d963c0b4427b927eaef51b952dc6 (127.4.200.67:42211): tablet f2034934a0fa4999bf3faf9127c44928 (table test-table [id=14249467f6a44345a540d25eea2896fa]) successfully deleted
I20260929 16:24:27.790746 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:27.790337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:34829 (local address 127.0.0.1:55390)
0929 16:24:27.790532 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:27.790634 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:34829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 16:24:27.790942 21929 catalog_manager.cc:4584] TS af34e12b03db46358225fe43928358d6 (127.4.200.65:34829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2034934a0fa4999bf3faf9127c44928: Network error: Client connection negotiation failed: client connection to 127.4.200.65:34829: connect: Connection refused (error 111)
I20260929 16:24:27.794063 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22274
I20260929 16:24:27.799513 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 21912
2026-09-29T16:24:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2334 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:27Z Disabled control of system clock
I20260929 16:24:27.830341 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33365
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:27.944717 22450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:27.945006 22450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:27.945042 22450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:27.949244 22450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:27.949361 22450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:27.949388 22450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:27.949419 22450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:27.949448 22450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:27.954464 22450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:33365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:27.955569 22450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:27.957011 22450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:27.966241 22455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.966326 22456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:27.966647 22458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:27.966917 22450 server_base.cc:1034] running on GCE node
I20260929 16:24:27.967664 22450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:27.969230 22450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:27.970492 22450 hybrid_clock.cc:648] HybridClock initialized: now 1790699067970468 us; error 40 us; skew 500 ppm
I20260929 16:24:27.972656 22450 webserver.cc:460] Webserver started at http://127.4.200.126:38383/ using document root <none> and password file <none>
I20260929 16:24:27.973292 22450 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:27.973388 22450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:27.973618 22450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:27.976229 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "9529a95344774ec0bc79dd0df9576b20"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "8c153020c39fa86e65a46525db4cf90d"
server_key_iv: "349ce86239893856da8552535aa6570c"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.976910 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "9529a95344774ec0bc79dd0df9576b20"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "8c153020c39fa86e65a46525db4cf90d"
server_key_iv: "349ce86239893856da8552535aa6570c"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.981472 22450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:27.984220 22464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:27.985628 22450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:27.985869 22450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "9529a95344774ec0bc79dd0df9576b20"
format_stamp: "Formatted at 2026-09-29 16:24:27 on dist-test-slave-s9ct"
server_key: "8c153020c39fa86e65a46525db4cf90d"
server_key_iv: "349ce86239893856da8552535aa6570c"
server_key_version: "encryptionkey@0"
I20260929 16:24:27.986044 22450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:28.000375 22450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:28.001176 22450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:28.001405 22450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:28.009390 22450 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:33365
I20260929 16:24:28.009410 22516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:33365 every 8 connection(s)
I20260929 16:24:28.010533 22450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:28.013656 22517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:28.017019 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22450
I20260929 16:24:28.017187 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:28.017493 4897 external_mini_cluster.cc:1428] Setting key a63f1a0ae9b582444f8e4f0ff166d327
I20260929 16:24:28.019860 22517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: Bootstrap starting.
I20260929 16:24:28.022687 22517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:28.023571 22517 log.cc:824] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:28.025913 22517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: No bootstrap required, opened a new log
I20260929 16:24:28.028728 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.019106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.4.200.126:33365)
0929 16:24:28.019809 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.019831 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.019878 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:24:28.020761 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.020786 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.021320 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.021705 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:28.022161 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.023700 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.024766 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.025450 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.025627 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.026623 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:28.026629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:28.026645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:28.027140 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:28.027210 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:28.027235 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:28.027337 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.027509 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.027793 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260929 16:24:28.030356 22517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:24:28.030645 22517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:28.030711 22517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9529a95344774ec0bc79dd0df9576b20, State: Initialized, Role: FOLLOWER
I20260929 16:24:28.031178 22517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:24:28.031347 22517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:28.031402 22517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:28.031473 22517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:28.032341 22517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:24:28.032621 22517 leader_election.cc:302] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9529a95344774ec0bc79dd0df9576b20; no voters:
I20260929 16:24:28.032861 22517 leader_election.cc:288] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:28.033020 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:28.033293 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [term 1 LEADER]: Becoming Leader. State: Replica: 9529a95344774ec0bc79dd0df9576b20, State: Running, Role: LEADER
I20260929 16:24:28.033632 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } }
I20260929 16:24:28.034068 22517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:28.035312 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9529a95344774ec0bc79dd0df9576b20. Latest consensus state: current_term: 1 leader_uuid: "9529a95344774ec0bc79dd0df9576b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } } }
I20260929 16:24:28.035463 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:28.035772 22523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9529a95344774ec0bc79dd0df9576b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9529a95344774ec0bc79dd0df9576b20" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 33365 } } }
I20260929 16:24:28.035847 22531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:28.035883 22523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:28.039196 22531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:28.044904 22531 catalog_manager.cc:1347] Generated new cluster ID: f0b9a7647ae045d59a20deba8157da92
I20260929 16:24:28.044974 22531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:28.062597 22531 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:28.063843 22531 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:28.077939 22531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: Generated new TSK 0
I20260929 16:24:28.078946 22531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:28.083798 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--builtin_ntp_servers=127.4.200.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:28.195355 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:28.195683 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:28.195786 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:28.200277 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:28.200357 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:28.200446 22541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:28.205507 22541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:28.206673 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:28.208110 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:28.218055 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:28.218056 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:28.218094 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:28.218894 22541 server_base.cc:1034] running on GCE node
I20260929 16:24:28.219357 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:28.220120 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:28.221331 22541 hybrid_clock.cc:648] HybridClock initialized: now 1790699068221310 us; error 35 us; skew 500 ppm
I20260929 16:24:28.223428 22541 webserver.cc:460] Webserver started at http://127.4.200.65:38741/ using document root <none> and password file <none>
I20260929 16:24:28.224021 22541 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:28.224117 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:28.224347 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:28.226485 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "38f49d0c93424b3e99e58e696ff76eac"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "c4919c833db23c50bed36219d90ce719"
server_key_iv: "68be90f6b3bcdcaa2123e1b8a1188415"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.227051 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "38f49d0c93424b3e99e58e696ff76eac"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "c4919c833db23c50bed36219d90ce719"
server_key_iv: "68be90f6b3bcdcaa2123e1b8a1188415"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.231962 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 16:24:28.234586 22555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.236049 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:24:28.236187 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "38f49d0c93424b3e99e58e696ff76eac"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "c4919c833db23c50bed36219d90ce719"
server_key_iv: "68be90f6b3bcdcaa2123e1b8a1188415"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.236306 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:28.253010 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:28.253695 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:28.253886 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:28.254513 22541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:28.255434 22541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:28.255515 22541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.255589 22541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:28.255625 22541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.264622 22541 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:33895
I20260929 16:24:28.264650 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:33895 every 8 connection(s)
I20260929 16:24:28.265544 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:28.270956 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22541
I20260929 16:24:28.271095 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:28.271374 4897 external_mini_cluster.cc:1428] Setting key eebbb6a91798167a94f94833f326cd33
I20260929 16:24:28.273300 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--builtin_ntp_servers=127.4.200.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:28.275247 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.267391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:40157 (local address 127.4.200.126:33365)
0929 16:24:28.267557 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.267562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.268595 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 16:24:28.269953 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.269955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.270003 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.270077 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:28.270907 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.271718 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.273006 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.273371 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.273491 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.274710 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:28.274717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:28.274721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:28.274739 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:28.274763 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:28.274776 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:28.274860 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.275001 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.275066 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:24:28.276001 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.267828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33365 (local address 127.4.200.65:40157)
0929 16:24:28.268451 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.268490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.269753 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.270201 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.270207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.270284 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:28.270769 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.270781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.271857 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.271864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.272887 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.272897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.273583 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.273588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.273768 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.274454 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:28.274456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:28.274496 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:28.274564 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:28.274865 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:28.274872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:28.274876 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.274967 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.275173 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260929 16:24:28.277225 22669 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33365
I20260929 16:24:28.277472 22669 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:28.278103 22669 heartbeater.cc:511] Master 127.4.200.126:33365 requested a full tablet report, sending...
I20260929 16:24:28.279722 22481 ts_manager.cc:194] Registered new tserver with Master: 38f49d0c93424b3e99e58e696ff76eac (127.4.200.65:33895)
I20260929 16:24:28.280867 22481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.65:40157
W20260929 16:24:28.382893 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:28.383127 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:28.383163 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:28.387756 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:28.387806 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:28.387897 22672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:28.392638 22672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:28.393636 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:28.395000 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:28.404529 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:28.404608 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:28.404600 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:28.404744 22672 server_base.cc:1034] running on GCE node
I20260929 16:24:28.405215 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:28.405836 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:28.407006 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790699068406991 us; error 27 us; skew 500 ppm
I20260929 16:24:28.408859 22672 webserver.cc:460] Webserver started at http://127.4.200.66:45791/ using document root <none> and password file <none>
I20260929 16:24:28.409446 22672 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:28.409508 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:28.409711 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:28.411775 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "92bd01bb5cac4ebf9d2ca035c1139830"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "d2f53173dc19837963e84ba106c8f4f8"
server_key_iv: "431b86fa029719c305b91c8af1c4322a"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.412266 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "92bd01bb5cac4ebf9d2ca035c1139830"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "d2f53173dc19837963e84ba106c8f4f8"
server_key_iv: "431b86fa029719c305b91c8af1c4322a"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.415807 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:24:28.418056 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.419348 22672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:28.419436 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "92bd01bb5cac4ebf9d2ca035c1139830"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "d2f53173dc19837963e84ba106c8f4f8"
server_key_iv: "431b86fa029719c305b91c8af1c4322a"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.419516 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:28.434439 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:28.435106 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:28.435364 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:28.435941 22672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:28.436898 22672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:28.436944 22672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.437011 22672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:28.437047 22672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.446087 22672 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:44969
I20260929 16:24:28.446120 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:44969 every 8 connection(s)
I20260929 16:24:28.447000 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:28.449797 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22672
I20260929 16:24:28.449919 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:28.450258 4897 external_mini_cluster.cc:1428] Setting key f8df1b59f633a95349c2618b2ce2ded2
I20260929 16:24:28.452395 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--builtin_ntp_servers=127.4.200.84:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:28.456997 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.448829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:48723 (local address 127.4.200.126:33365)
0929 16:24:28.448993 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.448998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.450248 (+ 1250us) server_negotiation.cc:407] Connection header received
0929 16:24:28.451945 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.451948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.451991 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.452059 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:28.453264 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.453892 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.455060 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.455281 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.455396 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.456501 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:28.456505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:28.456508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:28.456528 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:28.456547 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:28.456555 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:28.456631 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.456797 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.456858 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:24:28.457862 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.449246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33365 (local address 127.4.200.66:48723)
0929 16:24:28.450076 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.450134 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.451727 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.452268 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.452279 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.452386 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:28.453114 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.453136 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.454026 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.454028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.454934 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.454942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.455431 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.455435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.455620 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.456240 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:28.456241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:28.456279 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:28.456338 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:28.456648 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:28.456657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:28.456662 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.456786 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.457062 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":114,"threads_started":1}
I20260929 16:24:28.458977 22800 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33365
I20260929 16:24:28.459227 22800 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:28.459713 22800 heartbeater.cc:511] Master 127.4.200.126:33365 requested a full tablet report, sending...
I20260929 16:24:28.460741 22481 ts_manager.cc:194] Registered new tserver with Master: 92bd01bb5cac4ebf9d2ca035c1139830 (127.4.200.66:44969)
I20260929 16:24:28.461318 22481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.66:48723
W20260929 16:24:28.570135 22803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:28.570490 22803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:28.570567 22803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:28.574653 22803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:28.574744 22803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:28.574841 22803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:28.579758 22803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:28.580920 22803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:28.582329 22803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:28.592167 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:28.592186 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:28.592363 22803 server_base.cc:1034] running on GCE node
W20260929 16:24:28.592186 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:28.592824 22803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:28.593523 22803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:28.594703 22803 hybrid_clock.cc:648] HybridClock initialized: now 1790699068594686 us; error 28 us; skew 500 ppm
I20260929 16:24:28.596545 22803 webserver.cc:460] Webserver started at http://127.4.200.67:37325/ using document root <none> and password file <none>
I20260929 16:24:28.597136 22803 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:28.597235 22803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:28.597465 22803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:28.599655 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "fec2397e0a054df0bf9fe6d1cb835d54"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "2acc9d94858b645fb40becdb5858d187"
server_key_iv: "14f3aaebdc76375926a158bbf149e058"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.600258 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "fec2397e0a054df0bf9fe6d1cb835d54"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "2acc9d94858b645fb40becdb5858d187"
server_key_iv: "14f3aaebdc76375926a158bbf149e058"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.604386 22803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:24:28.606982 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.608292 22803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:28.608429 22803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "fec2397e0a054df0bf9fe6d1cb835d54"
format_stamp: "Formatted at 2026-09-29 16:24:28 on dist-test-slave-s9ct"
server_key: "2acc9d94858b645fb40becdb5858d187"
server_key_iv: "14f3aaebdc76375926a158bbf149e058"
server_key_version: "encryptionkey@0"
I20260929 16:24:28.608549 22803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:28.627543 22803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:28.628257 22803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:28.628455 22803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:28.629052 22803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:28.630004 22803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:28.630077 22803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.630148 22803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:28.630196 22803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:28.640012 22803 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:41259
I20260929 16:24:28.640076 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:41259 every 8 connection(s)
I20260929 16:24:28.641006 22803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:28.641911 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22803
I20260929 16:24:28.642026 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:28.642318 4897 external_mini_cluster.cc:1428] Setting key 00e6b7beafa14e759e21c6f17272fbad
I20260929 16:24:28.650718 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.643031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:36219 (local address 127.4.200.126:33365)
0929 16:24:28.643167 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.643170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.644336 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 16:24:28.645815 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.645818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.645873 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.645951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:28.646916 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.647459 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.648620 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.648870 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.648983 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.650151 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:28.650155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:28.650158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:28.650172 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:28.650201 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:28.650210 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:28.650287 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.650499 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.650568 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:24:28.651728 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.643525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:33365 (local address 127.4.200.67:36219)
0929 16:24:28.644201 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.644240 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.645613 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.646129 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.646136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.646224 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:28.646780 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.646793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.647610 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.647614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.648485 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.648495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.649026 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.649031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.649201 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.649863 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:28.649865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:28.649906 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:28.649975 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:28.650304 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:28.650313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:28.650319 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.650499 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.650809 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260929 16:24:28.652963 22931 heartbeater.cc:347] Connected to a master server at 127.4.200.126:33365
I20260929 16:24:28.653203 22931 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:28.653787 22931 heartbeater.cc:511] Master 127.4.200.126:33365 requested a full tablet report, sending...
I20260929 16:24:28.655002 22481 ts_manager.cc:194] Registered new tserver with Master: fec2397e0a054df0bf9fe6d1cb835d54 (127.4.200.67:41259)
I20260929 16:24:28.655659 22481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.67:36219
I20260929 16:24:28.656517 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:28.662745 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.658836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.4.200.126:33365)
0929 16:24:28.658997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.659001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.659189 (+ 188us) server_negotiation.cc:407] Connection header received
0929 16:24:28.659262 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.659264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.659307 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.659381 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:28.659721 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.660264 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.661142 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.661354 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.661437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.662261 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:28.662264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:28.662266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:28.662278 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:28.662297 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:28.662310 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:28.662370 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.662573 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.662631 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:24:28.666045 22481 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:28.669288 22481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:28.683880 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.678308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.4.200.66:44969)
0929 16:24:28.678754 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.678768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.679231 (+ 463us) server_negotiation.cc:407] Connection header received
0929 16:24:28.679409 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.679419 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.679625 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.679783 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:28.680049 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.680878 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.682332 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.683159 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.683268 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.683478 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.683595 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.683704 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":153,"threads_started":1}
I20260929 16:24:28.683883 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.677953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:44969 (local address 127.0.0.1:42576)
0929 16:24:28.679008 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.679060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.679208 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.679774 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.679780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.679801 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:28.679942 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.679949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.680995 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.681002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.682207 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.682216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.683302 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.683306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.683408 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.683445 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.683566 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.683724 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":3,"thread_start_us":125,"threads_started":1}
I20260929 16:24:28.684226 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.678994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.4.200.65:33895)
0929 16:24:28.679424 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.679434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.679456 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:24:28.679563 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.679567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.679729 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.679860 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:28.680195 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.681390 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.682666 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.683183 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.683314 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.683499 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.683955 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.684081 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":135,"threads_started":1}
I20260929 16:24:28.684451 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.678680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:33895 (local address 127.0.0.1:53200)
0929 16:24:28.679381 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.679394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.679465 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.679938 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.679941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.679956 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:28.680073 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.680085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.681526 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.681530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.682534 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.682544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.683564 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.683567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.683691 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.683712 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.683778 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.683833 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":253,"spinlock_wait_cycles":10624,"thread_start_us":92,"threads_started":1}
I20260929 16:24:28.685055 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.677960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:41259 (local address 127.0.0.1:58670)
0929 16:24:28.679059 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.679073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.679253 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.679950 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.679953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.679976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:28.680087 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.680097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.681526 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.681530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.683311 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.683321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.684622 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.684626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.684721 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.684751 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.684827 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.684913 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":590,"spinlock_wait_cycles":3328,"thread_start_us":69,"threads_started":1}
I20260929 16:24:28.685370 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.678100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.4.200.67:41259)
0929 16:24:28.678731 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.678745 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.679176 (+ 431us) server_negotiation.cc:407] Connection header received
0929 16:24:28.679416 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.679423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.679628 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.679755 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:28.680253 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.681389 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.683447 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.684226 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.684333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.684518 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.685083 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.685186 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":140,"threads_started":1}
I20260929 16:24:28.687448 22603 tablet_service.cc:1467] Processing CreateTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d (DEFAULT_TABLE table=test-table [id=df2f49d8cec141a5ab190c564bcda85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:28.687645 22734 tablet_service.cc:1467] Processing CreateTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d (DEFAULT_TABLE table=test-table [id=df2f49d8cec141a5ab190c564bcda85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:28.688494 22603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1723f914c6d4a36b7c13430b29d7f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:28.688702 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1723f914c6d4a36b7c13430b29d7f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:28.689069 22865 tablet_service.cc:1467] Processing CreateTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d (DEFAULT_TABLE table=test-table [id=df2f49d8cec141a5ab190c564bcda85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:28.690104 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1723f914c6d4a36b7c13430b29d7f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:28.694725 22948 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Bootstrap starting.
I20260929 16:24:28.696914 22948 tablet_bootstrap.cc:654] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:28.696995 22949 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Bootstrap starting.
I20260929 16:24:28.697862 22950 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Bootstrap starting.
I20260929 16:24:28.698230 22948 log.cc:824] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:28.699163 22949 tablet_bootstrap.cc:654] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:28.699944 22949 log.cc:824] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:28.700237 22950 tablet_bootstrap.cc:654] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:28.700446 22948 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: No bootstrap required, opened a new log
I20260929 16:24:28.700639 22948 ts_tablet_manager.cc:1397] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:24:28.701009 22950 log.cc:824] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:28.701875 22949 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: No bootstrap required, opened a new log
I20260929 16:24:28.702051 22949 ts_tablet_manager.cc:1397] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:28.702762 22950 tablet_bootstrap.cc:492] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: No bootstrap required, opened a new log
I20260929 16:24:28.702950 22950 ts_tablet_manager.cc:1397] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:28.705063 22948 raft_consensus.cc:348] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.705363 22948 raft_consensus.cc:374] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:28.705428 22948 raft_consensus.cc:729] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f49d0c93424b3e99e58e696ff76eac, State: Initialized, Role: FOLLOWER
I20260929 16:24:28.705916 22948 consensus_queue.cc:260] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.706102 22950 raft_consensus.cc:348] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.706363 22950 raft_consensus.cc:374] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:28.706234 22949 raft_consensus.cc:348] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.706449 22950 raft_consensus.cc:729] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bd01bb5cac4ebf9d2ca035c1139830, State: Initialized, Role: FOLLOWER
I20260929 16:24:28.706488 22949 raft_consensus.cc:374] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:28.706547 22949 raft_consensus.cc:729] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec2397e0a054df0bf9fe6d1cb835d54, State: Initialized, Role: FOLLOWER
I20260929 16:24:28.706866 22669 heartbeater.cc:503] Master 127.4.200.126:33365 was elected leader, sending a full tablet report...
I20260929 16:24:28.706897 22950 consensus_queue.cc:260] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.707007 22948 ts_tablet_manager.cc:1428] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:24:28.707471 22949 consensus_queue.cc:260] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.707849 22950 ts_tablet_manager.cc:1428] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:24:28.708184 22800 heartbeater.cc:503] Master 127.4.200.126:33365 was elected leader, sending a full tablet report...
I20260929 16:24:28.708338 22949 ts_tablet_manager.cc:1428] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:24:28.708721 22931 heartbeater.cc:503] Master 127.4.200.126:33365 was elected leader, sending a full tablet report...
W20260929 16:24:28.766685 22670 tablet.cc:2369] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:24:28.804559 22954 raft_consensus.cc:482] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:28.804848 22954 raft_consensus.cc:504] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.806084 22954 leader_election.cc:288] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec2397e0a054df0bf9fe6d1cb835d54 (127.4.200.67:41259), 92bd01bb5cac4ebf9d2ca035c1139830 (127.4.200.66:44969)
I20260929 16:24:28.810166 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.806528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:48431 (local address 127.4.200.67:41259)
0929 16:24:28.806689 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.806695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.806913 (+ 218us) server_negotiation.cc:407] Connection header received
0929 16:24:28.807075 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.807078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.807131 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.807220 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:28.807519 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.808045 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809134 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.809613 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809747 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.809801 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.809963 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.810040 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:24:28.810194 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.806414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:41259 (local address 127.4.200.65:48431)
0929 16:24:28.806794 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.806818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.806955 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.807232 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.807235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.807251 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:28.807406 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.807411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.808166 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.808171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.808985 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.808993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.809739 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.809829 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.809852 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.809947 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.810025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":17,"spinlock_wait_cycles":1280,"thread_start_us":74,"threads_started":1}
I20260929 16:24:28.810348 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.806728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:57195 (local address 127.4.200.66:44969)
0929 16:24:28.806887 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.806892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.806909 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:24:28.807076 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.807080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.807137 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.807235 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:28.807575 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.808117 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809242 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.809692 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809841 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.809893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.810113 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.810190 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:24:28.810837 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1723f914c6d4a36b7c13430b29d7f3d" candidate_uuid: "38f49d0c93424b3e99e58e696ff76eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" is_pre_election: true
I20260929 16:24:28.810920 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.806409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:44969 (local address 127.4.200.65:57195)
0929 16:24:28.806781 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.806802 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:28.806931 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:28.807247 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:28.807251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:28.807291 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:28.807449 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.807455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.808239 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.808245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.809108 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:28.809117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.809845 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.809849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:28.809962 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.809992 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:24:28.810096 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:24:28.810173 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":24,"spinlock_wait_cycles":23680,"thread_start_us":93,"threads_started":1}
I20260929 16:24:28.811177 22885 raft_consensus.cc:2454] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f49d0c93424b3e99e58e696ff76eac in term 0.
I20260929 16:24:28.811744 22557 leader_election.cc:302] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49d0c93424b3e99e58e696ff76eac, fec2397e0a054df0bf9fe6d1cb835d54; no voters:
I20260929 16:24:28.811723 22754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1723f914c6d4a36b7c13430b29d7f3d" candidate_uuid: "38f49d0c93424b3e99e58e696ff76eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" is_pre_election: true
I20260929 16:24:28.812023 22754 raft_consensus.cc:2454] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f49d0c93424b3e99e58e696ff76eac in term 0.
I20260929 16:24:28.812044 22954 raft_consensus.cc:2781] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:28.812121 22954 raft_consensus.cc:482] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:28.812175 22954 raft_consensus.cc:3037] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:28.813472 22954 raft_consensus.cc:504] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.813815 22954 leader_election.cc:288] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [CANDIDATE]: Term 1 election: Requested vote from peers fec2397e0a054df0bf9fe6d1cb835d54 (127.4.200.67:41259), 92bd01bb5cac4ebf9d2ca035c1139830 (127.4.200.66:44969)
I20260929 16:24:28.814153 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1723f914c6d4a36b7c13430b29d7f3d" candidate_uuid: "38f49d0c93424b3e99e58e696ff76eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec2397e0a054df0bf9fe6d1cb835d54"
I20260929 16:24:28.814172 22754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1723f914c6d4a36b7c13430b29d7f3d" candidate_uuid: "38f49d0c93424b3e99e58e696ff76eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bd01bb5cac4ebf9d2ca035c1139830"
I20260929 16:24:28.814293 22885 raft_consensus.cc:3037] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:28.814307 22754 raft_consensus.cc:3037] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:28.815374 22754 raft_consensus.cc:2454] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f49d0c93424b3e99e58e696ff76eac in term 1.
I20260929 16:24:28.815521 22885 raft_consensus.cc:2454] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f49d0c93424b3e99e58e696ff76eac in term 1.
I20260929 16:24:28.815733 22556 leader_election.cc:302] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49d0c93424b3e99e58e696ff76eac, 92bd01bb5cac4ebf9d2ca035c1139830; no voters:
I20260929 16:24:28.815901 22954 raft_consensus.cc:2781] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:28.816185 22954 raft_consensus.cc:686] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 1 LEADER]: Becoming Leader. State: Replica: 38f49d0c93424b3e99e58e696ff76eac, State: Running, Role: LEADER
I20260929 16:24:28.816529 22954 consensus_queue.cc:237] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } }
I20260929 16:24:28.819764 22481 catalog_manager.cc:5495] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f49d0c93424b3e99e58e696ff76eac (127.4.200.65). New cstate: current_term: 1 leader_uuid: "38f49d0c93424b3e99e58e696ff76eac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f49d0c93424b3e99e58e696ff76eac" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 33895 } health_report { overall_health: HEALTHY } } }
I20260929 16:24:28.837116 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.833415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.4.200.65:33895)
0929 16:24:28.833569 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:28.833573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:28.833586 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:24:28.833661 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:28.833663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:28.833728 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:28.833826 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:28.834237 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.834819 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.835750 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:28.835958 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:28.836106 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:28.836367 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:28.836795 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:28.836887 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:28.836941 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 16:24:28.836991 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:24:28.843233 22754 raft_consensus.cc:1261] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 1 FOLLOWER]: Refusing update from remote peer 38f49d0c93424b3e99e58e696ff76eac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:28.843235 22885 raft_consensus.cc:1261] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 1 FOLLOWER]: Refusing update from remote peer 38f49d0c93424b3e99e58e696ff76eac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:28.843983 22954 consensus_queue.cc:1035] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bd01bb5cac4ebf9d2ca035c1139830" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 44969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:28.844214 22959 consensus_queue.cc:1035] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec2397e0a054df0bf9fe6d1cb835d54" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:28.854041 22963 mvcc.cc:205] Tried to move back new op lower bound from 7334703385974648832 to 7334703385875099648. Current Snapshot: MvccSnapshot[applied={T|T < 7334703385974648832}]
I20260929 16:24:28.854247 22964 mvcc.cc:205] Tried to move back new op lower bound from 7334703385974648832 to 7334703385875099648. Current Snapshot: MvccSnapshot[applied={T|T < 7334703385974648832}]
I20260929 16:24:28.861232 22481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:28.861408 22481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:28.863755 22481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9529a95344774ec0bc79dd0df9576b20: Sending DeleteTablet for 3 replicas of tablet e1723f914c6d4a36b7c13430b29d7f3d
I20260929 16:24:28.864514 22865 tablet_service.cc:1514] Processing DeleteTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:28 UTC) from {username='slave'} at 127.0.0.1:58670
I20260929 16:24:28.864579 22603 tablet_service.cc:1514] Processing DeleteTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:28 UTC) from {username='slave'} at 127.0.0.1:53200
I20260929 16:24:28.865038 22734 tablet_service.cc:1514] Processing DeleteTablet for tablet e1723f914c6d4a36b7c13430b29d7f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:28 UTC) from {username='slave'} at 127.0.0.1:42576
I20260929 16:24:28.865504 22974 tablet_replica.cc:330] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: stopping tablet replica
I20260929 16:24:28.865713 22972 tablet_replica.cc:330] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac: stopping tablet replica
I20260929 16:24:28.865988 22972 raft_consensus.cc:2229] T e1723f914c6d4a36b7c13430b29d7f3d P 38f49d0c93424b3e99e58e696ff76eac [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:24:28.866209 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22541
I20260929 16:24:28.867875 22974 raft_consensus.cc:2229] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:28.868196 22974 raft_consensus.cc:2258] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:28.869733 22974 ts_tablet_manager.cc:1905] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:28.873312 22974 ts_tablet_manager.cc:1918] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:28.873461 22974 log.cc:1196] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/e1723f914c6d4a36b7c13430b29d7f3d
I20260929 16:24:28.873814 22974 ts_tablet_manager.cc:1939] T e1723f914c6d4a36b7c13430b29d7f3d P 92bd01bb5cac4ebf9d2ca035c1139830: Deleting consensus metadata
I20260929 16:24:28.874441 22973 tablet_replica.cc:330] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: stopping tablet replica
I20260929 16:24:28.874754 22973 raft_consensus.cc:2229] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:28.875118 22973 raft_consensus.cc:2258] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:28.875221 22465 catalog_manager.cc:4841] TS 92bd01bb5cac4ebf9d2ca035c1139830 (127.4.200.66:44969): tablet e1723f914c6d4a36b7c13430b29d7f3d (table test-table [id=df2f49d8cec141a5ab190c564bcda85f]) successfully deleted
W20260929 16:24:28.875880 22465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:33895 (error 108)
I20260929 16:24:28.876297 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22672
I20260929 16:24:28.878115 22973 ts_tablet_manager.cc:1905] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:28.879689 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:28.879194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:33895 (local address 127.0.0.1:53210)
0929 16:24:28.879500 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:28.879582 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:33895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260929 16:24:28.879971 22465 catalog_manager.cc:4584] TS 38f49d0c93424b3e99e58e696ff76eac (127.4.200.65:33895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1723f914c6d4a36b7c13430b29d7f3d: Network error: Client connection negotiation failed: client connection to 127.4.200.65:33895: connect: Connection refused (error 111)
I20260929 16:24:28.881021 22973 ts_tablet_manager.cc:1918] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:28.881101 22973 log.cc:1196] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/e1723f914c6d4a36b7c13430b29d7f3d
I20260929 16:24:28.881387 22973 ts_tablet_manager.cc:1939] T e1723f914c6d4a36b7c13430b29d7f3d P fec2397e0a054df0bf9fe6d1cb835d54: Deleting consensus metadata
I20260929 16:24:28.882541 22466 catalog_manager.cc:4841] TS fec2397e0a054df0bf9fe6d1cb835d54 (127.4.200.67:41259): tablet e1723f914c6d4a36b7c13430b29d7f3d (table test-table [id=df2f49d8cec141a5ab190c564bcda85f]) successfully deleted
I20260929 16:24:28.883525 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22803
I20260929 16:24:28.889050 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22450
2026-09-29T16:24:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:28 dist-test-slave-s9ct krb5kdc[22978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:28 dist-test-slave-s9ct krb5kdc[22978](info): set up 2 sockets
Sep 29 16:24:28 dist-test-slave-s9ct krb5kdc[22978](info): commencing operation
krb5kdc: starting...
W20260929 16:24:29.938310 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:30.062160 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:34387
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:34387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:30.177078 22991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:30.177532 22991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:30.177855 22991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:30.183933 22991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:30.184032 22991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:30.184069 22991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:30.184109 22991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:30.184137 22991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:30.192349 22991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:34387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:34387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:30.193492 22991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:30.194618 22991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:30.203121 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:30.203130 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.203431 22991 server_base.cc:1034] running on GCE node
W20260929 16:24:30.203621 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.204368 22991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:30.206127 22991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:30.207361 22991 hybrid_clock.cc:648] HybridClock initialized: now 1790699070207322 us; error 51 us; skew 500 ppm
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:30.211329 22991 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:30.212272 22991 webserver.cc:460] Webserver started at http://127.4.200.126:42099/ using document root <none> and password file <none>
I20260929 16:24:30.212877 22991 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:30.212973 22991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:30.213191 22991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:30.215420 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "78cba234e2924c2f8d19a434f5f1068f"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "0c52a01a43e78e77d90b06fe7e57b4d7"
server_key_iv: "284597644b07f29111dc446143ae35f1"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.216022 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "78cba234e2924c2f8d19a434f5f1068f"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "0c52a01a43e78e77d90b06fe7e57b4d7"
server_key_iv: "284597644b07f29111dc446143ae35f1"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.219882 22991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:30.222283 23006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.223495 22991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:30.223652 22991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "78cba234e2924c2f8d19a434f5f1068f"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "0c52a01a43e78e77d90b06fe7e57b4d7"
server_key_iv: "284597644b07f29111dc446143ae35f1"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.223768 22991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:30.244388 22991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:30.247268 22991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:30.247476 22991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:30.254670 22991 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:34387
I20260929 16:24:30.254674 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:34387 every 8 connection(s)
I20260929 16:24:30.255688 22991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:30.258319 23059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:30.258812 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 22991
I20260929 16:24:30.258982 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:30.259274 4897 external_mini_cluster.cc:1428] Setting key 26788a3069cda45df3212cd4547d9efd
I20260929 16:24:30.264950 23059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: Bootstrap starting.
I20260929 16:24:30.267491 23059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:30.268256 23059 log.cc:824] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:30.270349 23059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: No bootstrap required, opened a new log
I20260929 16:24:30.273530 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.260641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.4.200.126:34387)
0929 16:24:30.261044 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:30.261063 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:30.261105 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:24:30.261923 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:30.261944 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:30.262374 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:30.262762 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:30.263418 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.264832 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.265740 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.266457 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.266624 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.269177 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:30.269181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:30.269192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:30.269214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:30.270910 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.271382 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.271386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.271390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.271490 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.271781 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.271783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.271784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.272114 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:30.272246 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:30.272611 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:24:30.272890 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260929 16:24:30.274128 23059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } }
I20260929 16:24:30.274322 23059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:30.274379 23059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cba234e2924c2f8d19a434f5f1068f, State: Initialized, Role: FOLLOWER
I20260929 16:24:30.274848 23059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } }
I20260929 16:24:30.274997 23059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:30.275059 23059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:30.275164 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:30.276021 23059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } }
I20260929 16:24:30.276314 23059 leader_election.cc:302] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78cba234e2924c2f8d19a434f5f1068f; no voters:
I20260929 16:24:30.276539 23059 leader_election.cc:288] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:30.276706 23064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:30.276933 23064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [term 1 LEADER]: Becoming Leader. State: Replica: 78cba234e2924c2f8d19a434f5f1068f, State: Running, Role: LEADER
I20260929 16:24:30.277220 23064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } }
I20260929 16:24:30.277690 23059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:30.278870 23065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78cba234e2924c2f8d19a434f5f1068f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } } }
I20260929 16:24:30.278883 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78cba234e2924c2f8d19a434f5f1068f. Latest consensus state: current_term: 1 leader_uuid: "78cba234e2924c2f8d19a434f5f1068f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cba234e2924c2f8d19a434f5f1068f" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 34387 } } }
I20260929 16:24:30.279024 23065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:30.279043 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:30.279397 23073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:30.282413 23073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:30.288327 23073 catalog_manager.cc:1347] Generated new cluster ID: eb02a427d45741919bd5dd412b91f752
I20260929 16:24:30.288398 23073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:30.302627 23073 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:30.303556 23073 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:30.319931 23073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: Generated new TSK 0
I20260929 16:24:30.320650 23073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:24:30.354579 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:30.464401 23085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:30.464653 23085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:30.464694 23085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:30.468703 23085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:30.468760 23085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:30.468832 23085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:30.473914 23085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:30.474979 23085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:30.475937 23085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:30.485498 23093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:30.485531 23091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:30.485531 23090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.485796 23085 server_base.cc:1034] running on GCE node
I20260929 16:24:30.486184 23085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:30.486837 23085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:30.488019 23085 hybrid_clock.cc:648] HybridClock initialized: now 1790699070488001 us; error 28 us; skew 500 ppm
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:30.490653 23085 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:24:30.491441 23085 webserver.cc:460] Webserver started at http://127.4.200.65:41717/ using document root <none> and password file <none>
I20260929 16:24:30.491961 23085 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:30.492013 23085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:30.492165 23085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:30.494110 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "c5cd55ea94c04a39b567d3e5d535d7fc"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "4baeef44149f9f8d4a8487ac1837a83d"
server_key_iv: "cad2d58e0e721250d378ce7da72372fb"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.494678 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "c5cd55ea94c04a39b567d3e5d535d7fc"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "4baeef44149f9f8d4a8487ac1837a83d"
server_key_iv: "cad2d58e0e721250d378ce7da72372fb"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.498333 23085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:24:30.502728 23100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.504026 23085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 16:24:30.504144 23085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "c5cd55ea94c04a39b567d3e5d535d7fc"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "4baeef44149f9f8d4a8487ac1837a83d"
server_key_iv: "cad2d58e0e721250d378ce7da72372fb"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.504261 23085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:30.524654 23085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:30.534444 23085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:30.534663 23085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:30.535283 23085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:30.536120 23085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:30.536175 23085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.536245 23085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:30.536278 23085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.545390 23085 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:36785
I20260929 16:24:30.545410 23213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:36785 every 8 connection(s)
I20260929 16:24:30.546357 23085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:30.551237 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23085
I20260929 16:24:30.551383 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:30.551677 4897 external_mini_cluster.cc:1428] Setting key 6184c56e3eb5b5a760aead86321d8217
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:30.560703 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.548228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:55285 (local address 127.4.200.126:34387)
0929 16:24:30.548374 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:30.548378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:30.549188 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:24:30.550433 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:30.550441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:30.550488 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:30.550562 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:30.551903 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.552587 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.553700 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.553929 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.554038 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.556916 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:30.556920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:30.556922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:30.556945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:30.558528 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.559154 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.559156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.559158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.559230 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.559607 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.559609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.559610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.559767 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:30.559839 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:30.560480 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 16:24:30.560584 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:24:30.561926 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.548610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:34387 (local address 127.4.200.65:55285)
0929 16:24:30.549040 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:30.549082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:30.550203 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:30.550739 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:30.550746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:30.551098 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:30.551681 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.551697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.552712 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.552716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.553576 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.553586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.554084 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.554089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.554256 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.554926 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:30.554943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:30.556725 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:30.558645 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.558655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.558671 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.559057 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.559320 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.559324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.559326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.559516 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.559868 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:30.559877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:30.560226 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 16:24:30.560457 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:24:30.560816 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260929 16:24:30.563354 23214 heartbeater.cc:347] Connected to a master server at 127.4.200.126:34387
I20260929 16:24:30.563665 23214 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:30.564224 23214 heartbeater.cc:511] Master 127.4.200.126:34387 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
I20260929 16:24:30.566087 23023 ts_manager.cc:194] Registered new tserver with Master: c5cd55ea94c04a39b567d3e5d535d7fc (127.4.200.65:36785)
I20260929 16:24:30.567302 23023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:55285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:24:30.582054 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:30.688690 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:30.688982 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:30.689064 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:30.693230 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:30.693317 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:30.693439 23219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:30.698764 23219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:30.699843 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:30.700866 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:30.710574 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:30.710577 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.710831 23219 server_base.cc:1034] running on GCE node
W20260929 16:24:30.710574 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.711403 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:30.712067 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:30.713256 23219 hybrid_clock.cc:648] HybridClock initialized: now 1790699070713241 us; error 36 us; skew 500 ppm
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:30.716104 23219 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:24:30.716965 23219 webserver.cc:460] Webserver started at http://127.4.200.66:44347/ using document root <none> and password file <none>
I20260929 16:24:30.717573 23219 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:30.717633 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:30.717876 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:30.719995 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "3f66a105876f4c0eaefe82ee336494e6"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "7506a9d4bdc560367278bd432a5a0239"
server_key_iv: "3bb8be38468affdba5edc80f56128866"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.720556 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "3f66a105876f4c0eaefe82ee336494e6"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "7506a9d4bdc560367278bd432a5a0239"
server_key_iv: "3bb8be38468affdba5edc80f56128866"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.724243 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:24:30.726663 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.727957 23219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:24:30.728085 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "3f66a105876f4c0eaefe82ee336494e6"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "7506a9d4bdc560367278bd432a5a0239"
server_key_iv: "3bb8be38468affdba5edc80f56128866"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.728193 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:30.750627 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:30.753877 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:30.754069 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:30.754702 23219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:30.755498 23219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:30.755545 23219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.755585 23219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:30.755632 23219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.764531 23219 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:37867
I20260929 16:24:30.764554 23347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:37867 every 8 connection(s)
I20260929 16:24:30.765479 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:30.768224 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23219
I20260929 16:24:30.768383 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:30.768714 4897 external_mini_cluster.cc:1428] Setting key 5f2c83fe97ef4a1c5852976900702813
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:24:30.780722 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.767320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:33389 (local address 127.4.200.126:34387)
0929 16:24:30.767451 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:30.767456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:30.768341 (+ 885us) server_negotiation.cc:407] Connection header received
0929 16:24:30.769900 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:30.769903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:30.769944 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:30.770044 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:30.771278 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.772075 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.773812 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.774017 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.774097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.776907 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:30.776911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:30.776913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:30.776935 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:30.778502 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.779130 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.779135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.779137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.779221 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.779573 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.779578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.779579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.779855 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:30.779973 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:30.780445 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 16:24:30.780560 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:24:30.781498 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.767692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:34387 (local address 127.4.200.66:33389)
0929 16:24:30.768174 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:30.768235 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:30.769696 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:30.770236 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:30.770244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:30.770682 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:30.771134 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.771147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.772199 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.772204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.773685 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.773699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.774168 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.774175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.774433 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.775049 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:30.775074 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:30.776742 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:30.778635 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.778640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.778654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.779041 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.779336 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.779339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.779341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.779476 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.779971 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:30.779977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:30.780214 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 16:24:30.780402 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:24:30.780721 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:24:30.782691 23348 heartbeater.cc:347] Connected to a master server at 127.4.200.126:34387
I20260929 16:24:30.782953 23348 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:30.783517 23348 heartbeater.cc:511] Master 127.4.200.126:34387 requested a full tablet report, sending...
I20260929 16:24:30.784543 23023 ts_manager.cc:194] Registered new tserver with Master: 3f66a105876f4c0eaefe82ee336494e6 (127.4.200.66:37867)
I20260929 16:24:30.785106 23023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:33389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:24:30.797935 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:30.909395 23353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:30.909644 23353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:30.909687 23353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:30.913929 23353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:30.913990 23353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:30.914052 23353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:30.919325 23353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:30.920363 23353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:30.921343 23353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:30.931003 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:30.931054 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.931075 23353 server_base.cc:1034] running on GCE node
W20260929 16:24:30.931071 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:30.931748 23353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:30.932469 23353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:30.933663 23353 hybrid_clock.cc:648] HybridClock initialized: now 1790699070933652 us; error 37 us; skew 500 ppm
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:30.936507 23353 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:24:30.937330 23353 webserver.cc:460] Webserver started at http://127.4.200.67:32793/ using document root <none> and password file <none>
I20260929 16:24:30.937888 23353 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:30.937976 23353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:30.938220 23353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:30.940267 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "0249b026d0fa4f7a97d05d46774e714e"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "368ce5577515bc581dd304e8c12149b4"
server_key_iv: "e6f7a6e4d67e7eb0ea3042e369fa675f"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.940835 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "0249b026d0fa4f7a97d05d46774e714e"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "368ce5577515bc581dd304e8c12149b4"
server_key_iv: "e6f7a6e4d67e7eb0ea3042e369fa675f"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.944512 23353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 16:24:30.946996 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.948184 23353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:30.948311 23353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "0249b026d0fa4f7a97d05d46774e714e"
format_stamp: "Formatted at 2026-09-29 16:24:30 on dist-test-slave-s9ct"
server_key: "368ce5577515bc581dd304e8c12149b4"
server_key_iv: "e6f7a6e4d67e7eb0ea3042e369fa675f"
server_key_version: "encryptionkey@0"
I20260929 16:24:30.948415 23353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:30.962159 23353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:30.965541 23353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:30.965750 23353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:30.966342 23353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:30.967190 23353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:30.967260 23353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.967331 23353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:30.967374 23353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:30.975822 23353 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:35659
I20260929 16:24:30.975847 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:35659 every 8 connection(s)
I20260929 16:24:30.976703 23353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:30.984323 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23353
I20260929 16:24:30.984469 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:30.984769 4897 external_mini_cluster.cc:1428] Setting key 1ca6cf7d5f3f967237f92ec2eb0b639e
Sep 29 16:24:30 dist-test-slave-s9ct krb5kdc[22978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:30.991824 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.978850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:45037 (local address 127.4.200.126:34387)
0929 16:24:30.978991 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:30.978995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:30.980146 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 16:24:30.981682 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:30.981685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:30.981748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:30.981824 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:30.983106 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.983644 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.985258 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.985560 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.985691 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.988486 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:30.988489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:30.988491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:30.988518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:30.989978 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.990476 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.990479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.990480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.990554 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:30.990861 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:30.990863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:30.990864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:30.991041 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:30.991124 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:30.991604 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 16:24:30.991701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:24:30.992683 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:30.979345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:34387 (local address 127.4.200.67:45037)
0929 16:24:30.979949 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:30.980025 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:30.981447 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:30.982005 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:30.982013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:30.982379 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:30.982937 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.982951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.983774 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.983778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.985122 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:30.985131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:30.985788 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:30.985793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:30.985998 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:30.986731 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:30.986752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:30.988333 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:30.990079 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.990084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.990096 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.990344 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.990641 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:30.990643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:30.990644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:30.990776 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:30.991150 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:30.991155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:30.991395 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 16:24:30.991548 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:24:30.991809 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260929 16:24:30.993862 23482 heartbeater.cc:347] Connected to a master server at 127.4.200.126:34387
I20260929 16:24:30.994104 23482 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:30.994714 23482 heartbeater.cc:511] Master 127.4.200.126:34387 requested a full tablet report, sending...
I20260929 16:24:30.995743 23023 ts_manager.cc:194] Registered new tserver with Master: 0249b026d0fa4f7a97d05d46774e714e (127.4.200.67:35659)
I20260929 16:24:30.996317 23023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:45037
I20260929 16:24:30.999091 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:31.009539 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.001408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.4.200.126:34387)
0929 16:24:31.001584 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.001589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.001742 (+ 153us) server_negotiation.cc:407] Connection header received
0929 16:24:31.001899 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.001902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.001953 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.002041 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:31.002811 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.003335 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.004174 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.004394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.004501 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.006105 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:31.006110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:31.006113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:31.006149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:31.007570 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:31.008095 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:31.008099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:31.008101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:31.008195 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:31.008531 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:31.008535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:31.008536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:31.008754 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:31.008916 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.009271 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.009375 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:24:31.012712 23023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:31.015784 23023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:31.030627 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.024164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:35659 (local address 127.0.0.1:49864)
0929 16:24:31.025313 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.025325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.025415 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.025940 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.025944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.025966 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:31.026077 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.026085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027553 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.027560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.029241 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.029250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.030053 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.030166 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030202 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.030310 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.030449 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":279,"thread_start_us":101,"threads_started":1}
I20260929 16:24:31.030957 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.025124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.4.200.65:36785)
0929 16:24:31.025903 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.025920 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.025942 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:24:31.026028 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.026032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.026200 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.026351 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:31.026625 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027676 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.029172 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.030022 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030237 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030417 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.030572 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.030682 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":104,"threads_started":1}
I20260929 16:24:31.031090 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.024956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:36785 (local address 127.0.0.1:40450)
0929 16:24:31.025581 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.025595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.025665 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.026353 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.026357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.026379 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:31.026498 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.026505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027808 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.027812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.029053 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.029066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.030183 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.030428 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030449 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.030535 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.030593 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":114,"thread_start_us":98,"threads_started":1}
I20260929 16:24:31.031104 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.024164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37867 (local address 127.0.0.1:51876)
0929 16:24:31.024836 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.024884 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.025050 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.025682 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.025691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.025715 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:31.025866 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.025873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027580 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.027583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.029138 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.029148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.030183 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.030346 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030372 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.030468 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.030562 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":171,"thread_start_us":87,"threads_started":1}
I20260929 16:24:31.031394 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.024322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.4.200.67:35659)
0929 16:24:31.025099 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.025115 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.025429 (+ 314us) server_negotiation.cc:407] Connection header received
0929 16:24:31.025562 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.025569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.025767 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.026309 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:31.026506 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027416 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.029364 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.029924 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030054 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030213 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.030684 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.031032 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":110,"threads_started":1}
I20260929 16:24:31.031554 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.024523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.4.200.66:37867)
0929 16:24:31.024897 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.024911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.025169 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:24:31.025276 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.025281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.025509 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.026027 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:31.026176 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.027418 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.029254 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.030037 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.030712 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.030877 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.030930 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.031368 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260929 16:24:31.033947 23416 tablet_service.cc:1467] Processing CreateTablet for tablet 174279b2a905477995c9e6df56a5784c (DEFAULT_TABLE table=test-table [id=9256e3528e22419f934e80e351ae19fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:31.034004 23148 tablet_service.cc:1467] Processing CreateTablet for tablet 174279b2a905477995c9e6df56a5784c (DEFAULT_TABLE table=test-table [id=9256e3528e22419f934e80e351ae19fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:31.034917 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174279b2a905477995c9e6df56a5784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:31.034925 23416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174279b2a905477995c9e6df56a5784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:31.041072 23500 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Bootstrap starting.
I20260929 16:24:31.041836 23499 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: Bootstrap starting.
I20260929 16:24:31.043280 23500 tablet_bootstrap.cc:654] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:31.044180 23499 tablet_bootstrap.cc:654] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:31.043823 23282 tablet_service.cc:1467] Processing CreateTablet for tablet 174279b2a905477995c9e6df56a5784c (DEFAULT_TABLE table=test-table [id=9256e3528e22419f934e80e351ae19fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:31.044442 23500 log.cc:824] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:31.045033 23282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174279b2a905477995c9e6df56a5784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:31.045353 23499 log.cc:824] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:31.046538 23500 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: No bootstrap required, opened a new log
I20260929 16:24:31.046741 23500 ts_tablet_manager.cc:1397] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:31.047379 23499 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: No bootstrap required, opened a new log
I20260929 16:24:31.047570 23499 ts_tablet_manager.cc:1397] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:24:31.050756 23500 raft_consensus.cc:348] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.051059 23500 raft_consensus.cc:374] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:31.051123 23500 raft_consensus.cc:729] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0249b026d0fa4f7a97d05d46774e714e, State: Initialized, Role: FOLLOWER
I20260929 16:24:31.051573 23500 consensus_queue.cc:260] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.051374 23499 raft_consensus.cc:348] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.051728 23499 raft_consensus.cc:374] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:31.051792 23499 raft_consensus.cc:729] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cd55ea94c04a39b567d3e5d535d7fc, State: Initialized, Role: FOLLOWER
I20260929 16:24:31.051956 23503 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Bootstrap starting.
I20260929 16:24:31.052255 23499 consensus_queue.cc:260] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.052700 23500 ts_tablet_manager.cc:1428] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:31.052932 23214 heartbeater.cc:503] Master 127.4.200.126:34387 was elected leader, sending a full tablet report...
I20260929 16:24:31.053037 23482 heartbeater.cc:503] Master 127.4.200.126:34387 was elected leader, sending a full tablet report...
I20260929 16:24:31.053293 23499 ts_tablet_manager.cc:1428] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:24:31.055109 23503 tablet_bootstrap.cc:654] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:31.056031 23503 log.cc:824] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:31.058107 23503 tablet_bootstrap.cc:492] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: No bootstrap required, opened a new log
I20260929 16:24:31.058514 23503 ts_tablet_manager.cc:1397] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:24:31.061285 23503 raft_consensus.cc:348] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.061545 23503 raft_consensus.cc:374] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:31.061610 23503 raft_consensus.cc:729] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f66a105876f4c0eaefe82ee336494e6, State: Initialized, Role: FOLLOWER
I20260929 16:24:31.062008 23503 consensus_queue.cc:260] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.062721 23348 heartbeater.cc:503] Master 127.4.200.126:34387 was elected leader, sending a full tablet report...
I20260929 16:24:31.062933 23503 ts_tablet_manager.cc:1428] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:31.215050 23505 raft_consensus.cc:482] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:31.215243 23505 raft_consensus.cc:504] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.216333 23505 leader_election.cc:288] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0249b026d0fa4f7a97d05d46774e714e (127.4.200.67:35659), 3f66a105876f4c0eaefe82ee336494e6 (127.4.200.66:37867)
I20260929 16:24:31.220304 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.216705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:46993 (local address 127.4.200.66:37867)
0929 16:24:31.216877 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.216883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.217072 (+ 189us) server_negotiation.cc:407] Connection header received
0929 16:24:31.217220 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.217223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.217272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.217359 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:31.217647 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.218135 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219267 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.219678 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219811 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.219875 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.220097 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.220174 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:24:31.220301 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.216701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:39933 (local address 127.4.200.67:35659)
0929 16:24:31.216908 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.216914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.216958 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:24:31.217158 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.217160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.217203 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.217285 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:31.217584 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.218101 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219266 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.219677 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.219872 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.220084 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.220163 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260929 16:24:31.220341 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.216560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:35659 (local address 127.4.200.65:39933)
0929 16:24:31.216887 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.216901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.217035 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.217296 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.217299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.217315 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:31.217459 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.217465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.218211 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.218217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.219140 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.219150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.219803 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.219954 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.219974 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.220069 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.220175 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":114,"thread_start_us":117,"threads_started":1}
I20260929 16:24:31.220350 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.216615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:37867 (local address 127.4.200.65:46993)
0929 16:24:31.216979 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.216990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.217111 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.217367 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.217370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.217386 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:31.217538 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.217550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.218260 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.218264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.219145 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.219155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.219804 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.219807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.219973 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.219991 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.220082 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.220177 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":50,"thread_start_us":110,"threads_started":1}
I20260929 16:24:31.221258 23436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174279b2a905477995c9e6df56a5784c" candidate_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0249b026d0fa4f7a97d05d46774e714e" is_pre_election: true
I20260929 16:24:31.221397 23302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174279b2a905477995c9e6df56a5784c" candidate_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66a105876f4c0eaefe82ee336494e6" is_pre_election: true
I20260929 16:24:31.221596 23436 raft_consensus.cc:2454] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cd55ea94c04a39b567d3e5d535d7fc in term 0.
I20260929 16:24:31.221736 23302 raft_consensus.cc:2454] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cd55ea94c04a39b567d3e5d535d7fc in term 0.
I20260929 16:24:31.222110 23104 leader_election.cc:302] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0249b026d0fa4f7a97d05d46774e714e, c5cd55ea94c04a39b567d3e5d535d7fc; no voters:
I20260929 16:24:31.222450 23505 raft_consensus.cc:2781] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:31.222532 23505 raft_consensus.cc:482] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:31.222563 23505 raft_consensus.cc:3037] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:31.223609 23505 raft_consensus.cc:504] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
I20260929 16:24:31.223989 23505 leader_election.cc:288] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [CANDIDATE]: Term 1 election: Requested vote from peers 0249b026d0fa4f7a97d05d46774e714e (127.4.200.67:35659), 3f66a105876f4c0eaefe82ee336494e6 (127.4.200.66:37867)
I20260929 16:24:31.224326 23436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174279b2a905477995c9e6df56a5784c" candidate_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0249b026d0fa4f7a97d05d46774e714e"
I20260929 16:24:31.224344 23302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174279b2a905477995c9e6df56a5784c" candidate_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66a105876f4c0eaefe82ee336494e6"
I20260929 16:24:31.224459 23302 raft_consensus.cc:3037] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:31.224462 23436 raft_consensus.cc:3037] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:31.225488 23302 raft_consensus.cc:2454] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cd55ea94c04a39b567d3e5d535d7fc in term 1.
I20260929 16:24:31.225536 23436 raft_consensus.cc:2454] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cd55ea94c04a39b567d3e5d535d7fc in term 1.
I20260929 16:24:31.225848 23104 leader_election.cc:302] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0249b026d0fa4f7a97d05d46774e714e, c5cd55ea94c04a39b567d3e5d535d7fc; no voters:
I20260929 16:24:31.226037 23505 raft_consensus.cc:2781] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:31.226292 23505 raft_consensus.cc:686] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [term 1 LEADER]: Becoming Leader. State: Replica: c5cd55ea94c04a39b567d3e5d535d7fc, State: Running, Role: LEADER
I20260929 16:24:31.226651 23505 consensus_queue.cc:237] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } }
W20260929 16:24:31.227877 23483 tablet.cc:2369] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:24:31.229890 23022 catalog_manager.cc:5495] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cd55ea94c04a39b567d3e5d535d7fc (127.4.200.65). New cstate: current_term: 1 leader_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cd55ea94c04a39b567d3e5d535d7fc" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 36785 } health_report { overall_health: HEALTHY } } }
W20260929 16:24:31.266662 23349 tablet.cc:2369] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:24:31.269263 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.265474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.4.200.65:36785)
0929 16:24:31.265609 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.265613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.265652 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:24:31.265722 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.265724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.265780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.265859 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:31.266121 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.266769 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.267754 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.268094 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.268198 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.268485 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:31.268875 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:31.268987 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.269119 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.269155 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:24:31.275282 23302 raft_consensus.cc:1261] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 1 FOLLOWER]: Refusing update from remote peer c5cd55ea94c04a39b567d3e5d535d7fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:31.275283 23436 raft_consensus.cc:1261] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 1 FOLLOWER]: Refusing update from remote peer c5cd55ea94c04a39b567d3e5d535d7fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:31.275934 23505 consensus_queue.cc:1035] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f66a105876f4c0eaefe82ee336494e6" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:31.276136 23510 consensus_queue.cc:1035] T 174279b2a905477995c9e6df56a5784c P c5cd55ea94c04a39b567d3e5d535d7fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0249b026d0fa4f7a97d05d46774e714e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 35659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:31.292373 23022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:31.292567 23022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:31.294975 23022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78cba234e2924c2f8d19a434f5f1068f: Sending DeleteTablet for 3 replicas of tablet 174279b2a905477995c9e6df56a5784c
I20260929 16:24:31.295806 23148 tablet_service.cc:1514] Processing DeleteTablet for tablet 174279b2a905477995c9e6df56a5784c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:40450
I20260929 16:24:31.295861 23282 tablet_service.cc:1514] Processing DeleteTablet for tablet 174279b2a905477995c9e6df56a5784c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260929 16:24:31.295760 23416 tablet_service.cc:1514] Processing DeleteTablet for tablet 174279b2a905477995c9e6df56a5784c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:31 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:49864
I20260929 16:24:31.296320 23524 tablet_replica.cc:330] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: stopping tablet replica
I20260929 16:24:31.296626 23523 tablet_replica.cc:330] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: stopping tablet replica
I20260929 16:24:31.296630 23524 raft_consensus.cc:2229] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:31.296999 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23085
I20260929 16:24:31.296983 23523 raft_consensus.cc:2229] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:31.296995 23524 raft_consensus.cc:2258] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:31.298770 23523 raft_consensus.cc:2258] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:31.300128 23524 ts_tablet_manager.cc:1905] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:31.300313 23523 ts_tablet_manager.cc:1905] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:31.302477 23523 ts_tablet_manager.cc:1918] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:31.302589 23523 log.cc:1196] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/174279b2a905477995c9e6df56a5784c
I20260929 16:24:31.302872 23523 ts_tablet_manager.cc:1939] T 174279b2a905477995c9e6df56a5784c P 3f66a105876f4c0eaefe82ee336494e6: Deleting consensus metadata
I20260929 16:24:31.303624 23524 ts_tablet_manager.cc:1918] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:31.303728 23524 log.cc:1196] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/174279b2a905477995c9e6df56a5784c
I20260929 16:24:31.304077 23524 ts_tablet_manager.cc:1939] T 174279b2a905477995c9e6df56a5784c P 0249b026d0fa4f7a97d05d46774e714e: Deleting consensus metadata
I20260929 16:24:31.304039 23008 catalog_manager.cc:4841] TS 3f66a105876f4c0eaefe82ee336494e6 (127.4.200.66:37867): tablet 174279b2a905477995c9e6df56a5784c (table test-table [id=9256e3528e22419f934e80e351ae19fe]) successfully deleted
I20260929 16:24:31.305040 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23219
W20260929 16:24:31.305030 23008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:36785 (error 108)
I20260929 16:24:31.305461 23010 catalog_manager.cc:4841] TS 0249b026d0fa4f7a97d05d46774e714e (127.4.200.67:35659): tablet 174279b2a905477995c9e6df56a5784c (table test-table [id=9256e3528e22419f934e80e351ae19fe]) successfully deleted
I20260929 16:24:31.307431 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.307162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:36785 (local address 127.0.0.1:40466)
0929 16:24:31.307295 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.307361 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:36785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:24:31.307623 23008 catalog_manager.cc:4584] TS c5cd55ea94c04a39b567d3e5d535d7fc (127.4.200.65:36785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 174279b2a905477995c9e6df56a5784c: Network error: Client connection negotiation failed: client connection to 127.4.200.65:36785: connect: Connection refused (error 111)
I20260929 16:24:31.310989 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23353
I20260929 16:24:31.316520 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 22991
2026-09-29T16:24:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2427 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:31Z Disabled control of system clock
I20260929 16:24:31.348192 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:32885
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:31.465727 23529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:31.466037 23529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:31.466076 23529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:31.470244 23529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:31.470329 23529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:31.470345 23529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:31.470362 23529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:31.470378 23529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:31.475585 23529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:32885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:31.476755 23529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:31.478123 23529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:31.487492 23534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:31.487485 23537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:31.487485 23535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:31.487752 23529 server_base.cc:1034] running on GCE node
I20260929 16:24:31.488469 23529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:31.489665 23529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:31.490871 23529 hybrid_clock.cc:648] HybridClock initialized: now 1790699071490857 us; error 29 us; skew 500 ppm
I20260929 16:24:31.493049 23529 webserver.cc:460] Webserver started at http://127.4.200.126:42445/ using document root <none> and password file <none>
I20260929 16:24:31.493708 23529 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:31.493813 23529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:31.494022 23529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:31.496665 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "67fe20f68b6e49cfa526d72bf85decd4"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "34effcef059eb20ea99fa78b1585f4b4"
server_key_iv: "f7b0e74ad9eead46970d318532718255"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.497287 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "67fe20f68b6e49cfa526d72bf85decd4"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "34effcef059eb20ea99fa78b1585f4b4"
server_key_iv: "f7b0e74ad9eead46970d318532718255"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.501346 23529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:31.504361 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:31.505824 23529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:31.506100 23529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "67fe20f68b6e49cfa526d72bf85decd4"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "34effcef059eb20ea99fa78b1585f4b4"
server_key_iv: "f7b0e74ad9eead46970d318532718255"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.506242 23529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:31.529805 23529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:31.530715 23529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:31.530938 23529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:31.539469 23595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:32885 every 8 connection(s)
I20260929 16:24:31.539469 23529 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:32885
I20260929 16:24:31.540697 23529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:31.544124 23596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:31.545082 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23529
I20260929 16:24:31.545221 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:31.545569 4897 external_mini_cluster.cc:1428] Setting key 1ec5d6c52fb4982483b58da13fafde9e
I20260929 16:24:31.551970 23596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: Bootstrap starting.
I20260929 16:24:31.555910 23596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:31.556878 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.547293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.4.200.126:32885)
0929 16:24:31.548094 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.548115 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.548156 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:24:31.548833 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.548853 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.549567 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.550075 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:31.550587 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.551797 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.552893 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.553462 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.553674 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.554845 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:31.554851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:31.554868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:31.555333 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:31.555402 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:31.555424 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:31.555551 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.555704 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.556028 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":184,"threads_started":1}
I20260929 16:24:31.556851 23596 log.cc:824] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:31.559487 23596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: No bootstrap required, opened a new log
I20260929 16:24:31.563699 23596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } }
I20260929 16:24:31.564105 23596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:31.564169 23596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fe20f68b6e49cfa526d72bf85decd4, State: Initialized, Role: FOLLOWER
I20260929 16:24:31.564666 23596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } }
I20260929 16:24:31.564839 23596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:31.564913 23596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:31.565012 23596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:31.566051 23596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } }
I20260929 16:24:31.566512 23596 leader_election.cc:302] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67fe20f68b6e49cfa526d72bf85decd4; no voters:
I20260929 16:24:31.566876 23596 leader_election.cc:288] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:31.567016 23601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:31.567302 23601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [term 1 LEADER]: Becoming Leader. State: Replica: 67fe20f68b6e49cfa526d72bf85decd4, State: Running, Role: LEADER
I20260929 16:24:31.567633 23601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } }
I20260929 16:24:31.568233 23596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:31.569320 23603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67fe20f68b6e49cfa526d72bf85decd4. Latest consensus state: current_term: 1 leader_uuid: "67fe20f68b6e49cfa526d72bf85decd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } } }
I20260929 16:24:31.569468 23603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:31.569914 23610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:31.570159 23602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67fe20f68b6e49cfa526d72bf85decd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe20f68b6e49cfa526d72bf85decd4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 32885 } } }
I20260929 16:24:31.570255 23602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:31.572865 23610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:31.579749 23610 catalog_manager.cc:1347] Generated new cluster ID: 9ecb118fbc5545adb3c863a1732f9bc9
I20260929 16:24:31.579864 23610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:31.599280 23610 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:31.600248 23610 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:31.609251 23610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: Generated new TSK 0
I20260929 16:24:31.610103 23610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:31.615042 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--builtin_ntp_servers=127.4.200.84:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:31.739503 23620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:31.740365 23620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:31.740458 23620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:31.744925 23620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:31.745002 23620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:31.745074 23620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:31.750650 23620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:31.751866 23620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:31.753346 23620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:31.764469 23626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:31.764652 23620 server_base.cc:1034] running on GCE node
W20260929 16:24:31.764478 23628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:31.764691 23625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:31.765425 23620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:31.766106 23620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:31.767316 23620 hybrid_clock.cc:648] HybridClock initialized: now 1790699071767294 us; error 33 us; skew 500 ppm
I20260929 16:24:31.769461 23620 webserver.cc:460] Webserver started at http://127.4.200.65:45787/ using document root <none> and password file <none>
I20260929 16:24:31.770133 23620 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:31.770231 23620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:31.770500 23620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:31.772682 23620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "8f966aa0222242c387718d547262a54b"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "36466357758b136841b0adc21b6dd044"
server_key_iv: "d710ed4181c4e079901c7a5a9fb56b7c"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.773270 23620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "8f966aa0222242c387718d547262a54b"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "36466357758b136841b0adc21b6dd044"
server_key_iv: "d710ed4181c4e079901c7a5a9fb56b7c"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.777212 23620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:24:31.779943 23634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:31.781332 23620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:24:31.781483 23620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "8f966aa0222242c387718d547262a54b"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "36466357758b136841b0adc21b6dd044"
server_key_iv: "d710ed4181c4e079901c7a5a9fb56b7c"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.781620 23620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:31.809073 23620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:31.809924 23620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:31.810197 23620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:31.810986 23620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:31.812033 23620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:31.812109 23620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:31.812184 23620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:31.812232 23620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:31.821139 23620 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:40377
I20260929 16:24:31.821157 23747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:40377 every 8 connection(s)
I20260929 16:24:31.822274 23620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:31.823029 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23620
I20260929 16:24:31.823184 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:31.823477 4897 external_mini_cluster.cc:1428] Setting key 1c6c497d5fa139426b9a87e83147fa6e
I20260929 16:24:31.825717 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--builtin_ntp_servers=127.4.200.84:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:31.833891 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.824288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:35391 (local address 127.4.200.126:32885)
0929 16:24:31.824425 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:31.824431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:31.825881 (+ 1450us) server_negotiation.cc:407] Connection header received
0929 16:24:31.827940 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:31.827946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:31.828017 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:31.828135 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:31.829073 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.829683 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.831639 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.831866 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.831961 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.833395 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:31.833398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:31.833400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:31.833414 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:31.833432 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:31.833439 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:31.833495 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:31.833706 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 16:24:31.833769 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:24:31.835340 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:31.824904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:32885 (local address 127.4.200.65:35391)
0929 16:24:31.825716 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:31.825776 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:31.827701 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:31.828305 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:31.828312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:31.828399 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:31.828911 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.828924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.829836 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.829843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.831495 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:31.831511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:31.832050 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:31.832058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:31.832318 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:31.833031 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:31.833036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:31.833101 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:31.833215 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:31.833528 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:31.833540 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:31.833549 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:24:31.833707 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:24:31.834062 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260929 16:24:31.836829 23748 heartbeater.cc:347] Connected to a master server at 127.4.200.126:32885
I20260929 16:24:31.837096 23748 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:31.837659 23748 heartbeater.cc:511] Master 127.4.200.126:32885 requested a full tablet report, sending...
I20260929 16:24:31.839390 23559 ts_manager.cc:194] Registered new tserver with Master: 8f966aa0222242c387718d547262a54b (127.4.200.65:40377)
I20260929 16:24:31.840993 23559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.65:35391
W20260929 16:24:31.939460 23751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:31.939741 23751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:31.939780 23751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:31.943707 23751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:31.943750 23751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:31.943802 23751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:31.948401 23751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:31.949424 23751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:31.950841 23751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:31.960767 23756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:31.960760 23759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:31.960886 23751 server_base.cc:1034] running on GCE node
W20260929 16:24:31.961083 23757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:31.961575 23751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:31.962250 23751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:31.963469 23751 hybrid_clock.cc:648] HybridClock initialized: now 1790699071963433 us; error 70 us; skew 500 ppm
I20260929 16:24:31.965416 23751 webserver.cc:460] Webserver started at http://127.4.200.66:34497/ using document root <none> and password file <none>
I20260929 16:24:31.966024 23751 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:31.966127 23751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:31.966377 23751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:31.968626 23751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "c0f000468b2f4034882df51409a6c515"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "4dda9ec5b0989be87f7c5e09522fa790"
server_key_iv: "666a4ae7096d4e34a60e9040d88e442e"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.969218 23751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "c0f000468b2f4034882df51409a6c515"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "4dda9ec5b0989be87f7c5e09522fa790"
server_key_iv: "666a4ae7096d4e34a60e9040d88e442e"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.973055 23751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:24:31.975687 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:31.977030 23751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:31.977200 23751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "c0f000468b2f4034882df51409a6c515"
format_stamp: "Formatted at 2026-09-29 16:24:31 on dist-test-slave-s9ct"
server_key: "4dda9ec5b0989be87f7c5e09522fa790"
server_key_iv: "666a4ae7096d4e34a60e9040d88e442e"
server_key_version: "encryptionkey@0"
I20260929 16:24:31.977332 23751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:32.000262 23751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:32.000990 23751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:32.001206 23751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:32.001830 23751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:32.002943 23751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:32.003027 23751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:32.003103 23751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:32.003137 23751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:32.012458 23751 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:42561
I20260929 16:24:32.012482 23878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:42561 every 8 connection(s)
I20260929 16:24:32.013419 23751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:32.013689 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23751
I20260929 16:24:32.013837 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:32.014189 4897 external_mini_cluster.cc:1428] Setting key 67f0b4ef9ab2b1c25556742378058dba
I20260929 16:24:32.016479 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--builtin_ntp_servers=127.4.200.84:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:32.024396 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.015358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:47031 (local address 127.4.200.126:32885)
0929 16:24:32.015494 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.015498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.016893 (+ 1395us) server_negotiation.cc:407] Connection header received
0929 16:24:32.018818 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.018824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.018893 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.019006 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:32.019968 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.020912 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.022062 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.022517 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.022662 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.023787 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:32.023793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:32.023796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:32.023815 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:32.023844 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:32.023857 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:32.023948 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.024093 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.024178 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:24:32.025158 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.015904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:32885 (local address 127.4.200.66:47031)
0929 16:24:32.016709 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.016764 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.018602 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.019205 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.019212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.019302 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:32.019812 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.019825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.021055 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.021057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.021935 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.021944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.022695 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.022699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.022847 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.023510 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:32.023512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:32.023554 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:32.023617 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:32.023950 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:32.023956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:32.023960 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.024055 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.024280 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260929 16:24:32.026437 23879 heartbeater.cc:347] Connected to a master server at 127.4.200.126:32885
I20260929 16:24:32.026721 23879 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:32.027416 23879 heartbeater.cc:511] Master 127.4.200.126:32885 requested a full tablet report, sending...
I20260929 16:24:32.028556 23559 ts_manager.cc:194] Registered new tserver with Master: c0f000468b2f4034882df51409a6c515 (127.4.200.66:42561)
I20260929 16:24:32.029145 23559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.66:47031
W20260929 16:24:32.131896 23882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:32.132237 23882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:32.132315 23882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:32.136438 23882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:32.136547 23882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:32.136649 23882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:32.141654 23882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:32.142829 23882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:32.144259 23882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:32.154011 23888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:32.154088 23887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:32.154088 23890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:32.154210 23882 server_base.cc:1034] running on GCE node
I20260929 16:24:32.154791 23882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:32.155411 23882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:32.156615 23882 hybrid_clock.cc:648] HybridClock initialized: now 1790699072156590 us; error 32 us; skew 500 ppm
I20260929 16:24:32.158457 23882 webserver.cc:460] Webserver started at http://127.4.200.67:37387/ using document root <none> and password file <none>
I20260929 16:24:32.159029 23882 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:32.159111 23882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:32.159343 23882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:32.161356 23882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6"
format_stamp: "Formatted at 2026-09-29 16:24:32 on dist-test-slave-s9ct"
server_key: "fc924447b2e3dc4eff2a0dbd42c5660d"
server_key_iv: "02f34f807a1cd357bd1b687e40d7f5d4"
server_key_version: "encryptionkey@0"
I20260929 16:24:32.161909 23882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6"
format_stamp: "Formatted at 2026-09-29 16:24:32 on dist-test-slave-s9ct"
server_key: "fc924447b2e3dc4eff2a0dbd42c5660d"
server_key_iv: "02f34f807a1cd357bd1b687e40d7f5d4"
server_key_version: "encryptionkey@0"
I20260929 16:24:32.165509 23882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:24:32.168151 23896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:32.169579 23882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:24:32.169713 23882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6"
format_stamp: "Formatted at 2026-09-29 16:24:32 on dist-test-slave-s9ct"
server_key: "fc924447b2e3dc4eff2a0dbd42c5660d"
server_key_iv: "02f34f807a1cd357bd1b687e40d7f5d4"
server_key_version: "encryptionkey@0"
I20260929 16:24:32.169852 23882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:32.214589 23882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:32.215368 23882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:32.215605 23882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:32.216269 23882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:32.217290 23882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:32.217342 23882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:32.217422 23882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:32.217470 23882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:32.227147 23882 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:45639
I20260929 16:24:32.227175 24009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:45639 every 8 connection(s)
I20260929 16:24:32.228127 23882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:32.233938 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 23882
I20260929 16:24:32.234037 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:32.234323 4897 external_mini_cluster.cc:1428] Setting key d6b86e6d98c9f664d500279768ef4c27
I20260929 16:24:32.237797 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.230066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:47913 (local address 127.4.200.126:32885)
0929 16:24:32.230233 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.230238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.231337 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 16:24:32.232649 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.232652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.232699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.232774 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:32.233652 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.234266 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.235884 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.236121 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.236259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.237391 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:32.237394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:32.237396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:32.237408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:32.237426 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:32.237434 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:32.237495 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.237638 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.237688 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:24:32.238765 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.230552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:32885 (local address 127.4.200.67:47913)
0929 16:24:32.231189 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.231229 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.232456 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.232935 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.232942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.233022 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:32.233495 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.233507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.234421 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.234425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.235743 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.235754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.236280 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.236285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.236488 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.237149 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:32.237150 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:32.237188 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:32.237247 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:32.237508 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:32.237514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:32.237518 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.237615 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.237830 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260929 16:24:32.239952 24010 heartbeater.cc:347] Connected to a master server at 127.4.200.126:32885
I20260929 16:24:32.240160 24010 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:32.240613 24010 heartbeater.cc:511] Master 127.4.200.126:32885 requested a full tablet report, sending...
I20260929 16:24:32.241632 23559 ts_manager.cc:194] Registered new tserver with Master: a3bce9a0ffaf4bc9a1e2eb92b2b921f6 (127.4.200.67:45639)
I20260929 16:24:32.242220 23559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.67:47913
I20260929 16:24:32.248085 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:32.253799 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.250018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.4.200.126:32885)
0929 16:24:32.250165 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.250169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.250332 (+ 163us) server_negotiation.cc:407] Connection header received
0929 16:24:32.250486 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.250489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.250527 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.250608 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:32.250835 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.251330 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.252182 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.252385 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.252467 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.253382 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:32.253385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:32.253387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:32.253397 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:32.253415 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:32.253423 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:32.253522 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.253655 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.253688 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:24:32.256781 23559 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:32.259918 23559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:32.278936 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.268385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:45639 (local address 127.0.0.1:39844)
0929 16:24:32.269428 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.269442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.269516 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.270174 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.270182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.270207 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:32.270367 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.270374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.275017 (+ 4643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.275020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.277315 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.277325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278290 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.278444 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278490 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.278610 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.278758 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":154,"thread_start_us":91,"threads_started":1}
I20260929 16:24:32.278968 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.269032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.4.200.67:45639)
0929 16:24:32.269751 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.269762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.269780 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:24:32.269842 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.269846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.269991 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.270530 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:32.270676 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.274901 (+ 4225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.277452 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278145 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278260 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278505 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.278654 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.278782 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":103,"threads_started":1}
I20260929 16:24:32.279471 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.269099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.4.200.65:40377)
0929 16:24:32.269726 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.269738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.269763 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:24:32.269839 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.269845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.270037 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.270193 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:32.270993 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.276381 (+ 5388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.277723 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278341 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278464 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278658 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.279216 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.279320 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":113,"threads_started":1}
I20260929 16:24:32.279590 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.268222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.4.200.66:42561)
0929 16:24:32.269637 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.269651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.269677 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:24:32.269760 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.269765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.269991 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.270245 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:32.270643 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.274842 (+ 4199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.277272 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278142 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278505 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.279334 (+ 829us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.279442 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":154,"threads_started":1}
I20260929 16:24:32.279119 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.268128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:40377 (local address 127.0.0.1:40222)
0929 16:24:32.269229 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.269243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.269342 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.270779 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.270781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.270797 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:32.270863 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.270868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.276515 (+ 5647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.276518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.277592 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.277600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278718 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.278828 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278848 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.278915 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.278987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260929 16:24:32.279899 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.268081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:42561 (local address 127.0.0.1:45536)
0929 16:24:32.269053 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.269103 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.269259 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.270377 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.270380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.270417 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:32.270509 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.270525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.274999 (+ 4474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.275006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.277109 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.277119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.278610 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.278616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.278708 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.278769 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.278842 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.278918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":520,"thread_start_us":108,"threads_started":1}
I20260929 16:24:32.282881 23682 tablet_service.cc:1467] Processing CreateTablet for tablet c97b02d4df584407aea0599a13aa98be (DEFAULT_TABLE table=test-table [id=16384455e4d04d9281e4e490edddf65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:32.282874 23944 tablet_service.cc:1467] Processing CreateTablet for tablet c97b02d4df584407aea0599a13aa98be (DEFAULT_TABLE table=test-table [id=16384455e4d04d9281e4e490edddf65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:32.282881 23813 tablet_service.cc:1467] Processing CreateTablet for tablet c97b02d4df584407aea0599a13aa98be (DEFAULT_TABLE table=test-table [id=16384455e4d04d9281e4e490edddf65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:32.284025 23813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b02d4df584407aea0599a13aa98be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:32.284025 23682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b02d4df584407aea0599a13aa98be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:32.284026 23944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97b02d4df584407aea0599a13aa98be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:32.290704 24027 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Bootstrap starting.
I20260929 16:24:32.292059 24028 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Bootstrap starting.
I20260929 16:24:32.292059 24029 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Bootstrap starting.
I20260929 16:24:32.292996 24027 tablet_bootstrap.cc:654] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:32.293742 24027 log.cc:824] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:32.294236 24029 tablet_bootstrap.cc:654] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:32.294236 24028 tablet_bootstrap.cc:654] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:32.295120 24029 log.cc:824] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:32.295110 24028 log.cc:824] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:32.296233 24027 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: No bootstrap required, opened a new log
I20260929 16:24:32.296443 24027 ts_tablet_manager.cc:1397] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:24:32.297063 24029 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: No bootstrap required, opened a new log
I20260929 16:24:32.297065 24028 tablet_bootstrap.cc:492] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: No bootstrap required, opened a new log
I20260929 16:24:32.297250 24029 ts_tablet_manager.cc:1397] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:32.297250 24028 ts_tablet_manager.cc:1397] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:32.299795 24027 raft_consensus.cc:348] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.300047 24027 raft_consensus.cc:374] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:32.300109 24027 raft_consensus.cc:729] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bce9a0ffaf4bc9a1e2eb92b2b921f6, State: Initialized, Role: FOLLOWER
I20260929 16:24:32.300556 24027 consensus_queue.cc:260] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.301409 24010 heartbeater.cc:503] Master 127.4.200.126:32885 was elected leader, sending a full tablet report...
I20260929 16:24:32.301477 24029 raft_consensus.cc:348] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.301477 24028 raft_consensus.cc:348] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.301712 24028 raft_consensus.cc:374] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:32.301712 24029 raft_consensus.cc:374] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:32.301761 24029 raft_consensus.cc:729] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f966aa0222242c387718d547262a54b, State: Initialized, Role: FOLLOWER
I20260929 16:24:32.301771 24028 raft_consensus.cc:729] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f000468b2f4034882df51409a6c515, State: Initialized, Role: FOLLOWER
I20260929 16:24:32.302227 24028 consensus_queue.cc:260] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.302227 24029 consensus_queue.cc:260] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.302485 24027 ts_tablet_manager.cc:1428] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:24:32.303193 24028 ts_tablet_manager.cc:1428] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:24:32.303194 24029 ts_tablet_manager.cc:1428] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:24:32.303555 23879 heartbeater.cc:503] Master 127.4.200.126:32885 was elected leader, sending a full tablet report...
I20260929 16:24:32.303746 23748 heartbeater.cc:503] Master 127.4.200.126:32885 was elected leader, sending a full tablet report...
W20260929 16:24:32.323675 23749 tablet.cc:2369] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:24:32.479303 24011 tablet.cc:2369] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:24:32.514621 23880 tablet.cc:2369] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:24:32.527927 24035 raft_consensus.cc:482] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:32.528182 24035 raft_consensus.cc:504] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.529407 24035 leader_election.cc:288] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f966aa0222242c387718d547262a54b (127.4.200.65:40377), a3bce9a0ffaf4bc9a1e2eb92b2b921f6 (127.4.200.67:45639)
I20260929 16:24:32.533452 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.529755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:51609 (local address 127.4.200.65:40377)
0929 16:24:32.529906 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.529910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.529995 (+ 85us) server_negotiation.cc:407] Connection header received
0929 16:24:32.530186 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.530189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.530248 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.530333 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:32.530671 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.531206 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532443 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.532833 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532974 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.533023 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.533226 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.533303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:24:32.533473 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.529761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:45639 (local address 127.4.200.66:39847)
0929 16:24:32.530033 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.530061 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.530191 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.530495 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.530497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.530512 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:32.530647 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.530656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.531372 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.531374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.532304 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.532313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.532974 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.533111 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.533129 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.533222 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.533303 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":24,"thread_start_us":82,"threads_started":1}
I20260929 16:24:32.533463 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.529866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:39847 (local address 127.4.200.67:45639)
0929 16:24:32.529995 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.529999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.530170 (+ 171us) server_negotiation.cc:407] Connection header received
0929 16:24:32.530280 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.530282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.530355 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.530458 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:32.530759 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.531234 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532435 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.532833 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532960 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.533012 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.533258 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.533325 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:24:32.533483 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.529602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:40377 (local address 127.4.200.66:51609)
0929 16:24:32.529903 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.529922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:32.530049 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:32.530356 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:32.530359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:32.530378 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:32.530535 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.530544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.531326 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.531333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.532308 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:32.532317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.532960 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.532964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:32.533088 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.533117 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:24:32.533176 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 16:24:32.533278 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":89,"thread_start_us":70,"threads_started":1}
I20260929 16:24:32.534472 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b02d4df584407aea0599a13aa98be" candidate_uuid: "c0f000468b2f4034882df51409a6c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f966aa0222242c387718d547262a54b" is_pre_election: true
I20260929 16:24:32.534471 23964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b02d4df584407aea0599a13aa98be" candidate_uuid: "c0f000468b2f4034882df51409a6c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" is_pre_election: true
I20260929 16:24:32.534762 23964 raft_consensus.cc:2454] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f000468b2f4034882df51409a6c515 in term 0.
I20260929 16:24:32.534775 23702 raft_consensus.cc:2454] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f000468b2f4034882df51409a6c515 in term 0.
I20260929 16:24:32.535239 23767 leader_election.cc:302] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f966aa0222242c387718d547262a54b, c0f000468b2f4034882df51409a6c515; no voters:
I20260929 16:24:32.535522 24035 raft_consensus.cc:2781] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:32.535609 24035 raft_consensus.cc:482] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:32.535658 24035 raft_consensus.cc:3037] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:32.536669 24035 raft_consensus.cc:504] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.537063 24035 leader_election.cc:288] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [CANDIDATE]: Term 1 election: Requested vote from peers 8f966aa0222242c387718d547262a54b (127.4.200.65:40377), a3bce9a0ffaf4bc9a1e2eb92b2b921f6 (127.4.200.67:45639)
I20260929 16:24:32.537436 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b02d4df584407aea0599a13aa98be" candidate_uuid: "c0f000468b2f4034882df51409a6c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f966aa0222242c387718d547262a54b"
I20260929 16:24:32.537563 23702 raft_consensus.cc:3037] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:32.538641 23964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97b02d4df584407aea0599a13aa98be" candidate_uuid: "c0f000468b2f4034882df51409a6c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6"
I20260929 16:24:32.538775 23702 raft_consensus.cc:2454] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f000468b2f4034882df51409a6c515 in term 1.
I20260929 16:24:32.538790 23964 raft_consensus.cc:3037] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:32.539090 23767 leader_election.cc:302] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f966aa0222242c387718d547262a54b, c0f000468b2f4034882df51409a6c515; no voters:
I20260929 16:24:32.539297 24035 raft_consensus.cc:2781] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:32.539556 24035 raft_consensus.cc:686] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 1 LEADER]: Becoming Leader. State: Replica: c0f000468b2f4034882df51409a6c515, State: Running, Role: LEADER
I20260929 16:24:32.540064 23964 raft_consensus.cc:2454] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f000468b2f4034882df51409a6c515 in term 1.
I20260929 16:24:32.540112 24035 consensus_queue.cc:237] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } }
I20260929 16:24:32.543581 23555 catalog_manager.cc:5495] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f000468b2f4034882df51409a6c515 (127.4.200.66). New cstate: current_term: 1 leader_uuid: "c0f000468b2f4034882df51409a6c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f000468b2f4034882df51409a6c515" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 42561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 } health_report { overall_health: UNKNOWN } } }
I20260929 16:24:32.573083 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.569388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.4.200.66:42561)
0929 16:24:32.569528 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:32.569538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:32.569553 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:24:32.569693 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:32.569695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:32.569753 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:32.569847 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:32.570078 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.570713 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.571547 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:32.571761 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:32.571846 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:32.572161 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:32.572570 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:32.572680 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:32.572825 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:24:32.572896 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:24:32.579047 23702 raft_consensus.cc:1261] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b [term 1 FOLLOWER]: Refusing update from remote peer c0f000468b2f4034882df51409a6c515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:32.579047 23964 raft_consensus.cc:1261] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 1 FOLLOWER]: Refusing update from remote peer c0f000468b2f4034882df51409a6c515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:32.579802 24038 consensus_queue.cc:1035] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f966aa0222242c387718d547262a54b" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:32.579978 24035 consensus_queue.cc:1035] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bce9a0ffaf4bc9a1e2eb92b2b921f6" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 45639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:32.595734 23555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:32.596006 23555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:32.598733 23555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67fe20f68b6e49cfa526d72bf85decd4: Sending DeleteTablet for 3 replicas of tablet c97b02d4df584407aea0599a13aa98be
I20260929 16:24:32.599534 23813 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b02d4df584407aea0599a13aa98be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:32 UTC) from {username='slave'} at 127.0.0.1:45536
I20260929 16:24:32.599810 23682 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b02d4df584407aea0599a13aa98be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:32 UTC) from {username='slave'} at 127.0.0.1:40222
I20260929 16:24:32.600204 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23620
I20260929 16:24:32.600282 23944 tablet_service.cc:1514] Processing DeleteTablet for tablet c97b02d4df584407aea0599a13aa98be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:32 UTC) from {username='slave'} at 127.0.0.1:39844
I20260929 16:24:32.600359 24051 tablet_replica.cc:330] T c97b02d4df584407aea0599a13aa98be P 8f966aa0222242c387718d547262a54b: stopping tablet replica
I20260929 16:24:32.601071 24050 tablet_replica.cc:330] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: stopping tablet replica
I20260929 16:24:32.601394 24050 raft_consensus.cc:2229] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:24:32.601806 24050 raft_consensus.cc:2258] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:32.603482 24050 ts_tablet_manager.cc:1905] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:32.605794 24050 ts_tablet_manager.cc:1918] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:32.605911 24050 log.cc:1196] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/c97b02d4df584407aea0599a13aa98be
I20260929 16:24:32.606199 24050 ts_tablet_manager.cc:1939] T c97b02d4df584407aea0599a13aa98be P c0f000468b2f4034882df51409a6c515: Deleting consensus metadata
I20260929 16:24:32.607292 23546 catalog_manager.cc:4841] TS c0f000468b2f4034882df51409a6c515 (127.4.200.66:42561): tablet c97b02d4df584407aea0599a13aa98be (table test-table [id=16384455e4d04d9281e4e490edddf65f]) successfully deleted
W20260929 16:24:32.609592 23545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:40377 (error 108)
I20260929 16:24:32.609962 24052 tablet_replica.cc:330] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: stopping tablet replica
I20260929 16:24:32.610299 24052 raft_consensus.cc:2229] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:32.610697 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:32.610368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:40377 (local address 127.0.0.1:40238)
0929 16:24:32.610554 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:32.610616 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:40377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:24:32.610733 24052 raft_consensus.cc:2258] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:24:32.610844 23545 catalog_manager.cc:4584] TS 8f966aa0222242c387718d547262a54b (127.4.200.65:40377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c97b02d4df584407aea0599a13aa98be: Network error: Client connection negotiation failed: client connection to 127.4.200.65:40377: connect: Connection refused (error 111)
I20260929 16:24:32.611197 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23751
I20260929 16:24:32.613978 24052 ts_tablet_manager.cc:1905] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:32.616127 24052 ts_tablet_manager.cc:1918] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:32.616225 24052 log.cc:1196] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/c97b02d4df584407aea0599a13aa98be
I20260929 16:24:32.616515 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23882
I20260929 16:24:32.616540 24052 ts_tablet_manager.cc:1939] T c97b02d4df584407aea0599a13aa98be P a3bce9a0ffaf4bc9a1e2eb92b2b921f6: Deleting consensus metadata
I20260929 16:24:32.622223 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 23529
2026-09-29T16:24:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1304 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:32 dist-test-slave-s9ct krb5kdc[24056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:32 dist-test-slave-s9ct krb5kdc[24056](info): set up 2 sockets
Sep 29 16:24:32 dist-test-slave-s9ct krb5kdc[24056](info): commencing operation
krb5kdc: starting...
W20260929 16:24:33.662632 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:33 dist-test-slave-s9ct krb5kdc[24056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:33.792090 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:42941
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:42941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:33.907452 24069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:33.907783 24069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:33.907868 24069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:33.912050 24069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:33.912146 24069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:33.912220 24069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:33.912278 24069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:33.912315 24069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:33.917833 24069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:42941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:33.919034 24069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:33.920106 24069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:33.928774 24075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:33.928753 24077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:33.928844 24069 server_base.cc:1034] running on GCE node
W20260929 16:24:33.928778 24074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:33.929507 24069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:33.930684 24069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:33.931898 24069 hybrid_clock.cc:648] HybridClock initialized: now 1790699073931867 us; error 41 us; skew 500 ppm
Sep 29 16:24:33 dist-test-slave-s9ct krb5kdc[24056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:33.934755 24069 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:33.935636 24069 webserver.cc:460] Webserver started at http://127.4.200.126:34535/ using document root <none> and password file <none>
I20260929 16:24:33.936205 24069 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:33.936300 24069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:33.936511 24069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:33.938632 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "75b52da3303c4b4089b6a12b6b92a593"
format_stamp: "Formatted at 2026-09-29 16:24:33 on dist-test-slave-s9ct"
server_key: "1aa8186a3888e1fe6eb459462c683ff1"
server_key_iv: "cd51154e2c6dcd5d3d708602c294b633"
server_key_version: "encryptionkey@0"
I20260929 16:24:33.939194 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "75b52da3303c4b4089b6a12b6b92a593"
format_stamp: "Formatted at 2026-09-29 16:24:33 on dist-test-slave-s9ct"
server_key: "1aa8186a3888e1fe6eb459462c683ff1"
server_key_iv: "cd51154e2c6dcd5d3d708602c294b633"
server_key_version: "encryptionkey@0"
I20260929 16:24:33.942978 24069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:33.945539 24084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:33.946919 24069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:33.947075 24069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "75b52da3303c4b4089b6a12b6b92a593"
format_stamp: "Formatted at 2026-09-29 16:24:33 on dist-test-slave-s9ct"
server_key: "1aa8186a3888e1fe6eb459462c683ff1"
server_key_iv: "cd51154e2c6dcd5d3d708602c294b633"
server_key_version: "encryptionkey@0"
I20260929 16:24:33.947197 24069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:33.965833 24069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:33.968878 24069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:33.969094 24069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:33.976285 24069 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:42941
I20260929 16:24:33.976311 24136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:42941 every 8 connection(s)
I20260929 16:24:33.977231 24069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:33.978986 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24069
I20260929 16:24:33.979148 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:33.979434 4897 external_mini_cluster.cc:1428] Setting key 3082324012a2cbd4449e736c064215db
I20260929 16:24:33.980305 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:33.987422 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: Bootstrap starting.
Sep 29 16:24:33 dist-test-slave-s9ct krb5kdc[24056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:33.990080 24137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:33.991111 24137 log.cc:824] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:33.993798 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: No bootstrap required, opened a new log
I20260929 16:24:33.996387 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:33.980967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.4.200.126:42941)
0929 16:24:33.981400 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:33.981419 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:33.981461 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:24:33.982348 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:33.982373 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:33.982852 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:33.983245 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:33.984045 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:33.985557 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:33.986538 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:33.987211 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:33.987375 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:33.990253 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:33.990260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:33.990278 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:33.990317 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:33.992713 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:33.993265 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:33.993271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:33.993277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:33.993413 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:33.993736 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:33.993739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:33.993741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:33.994272 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:33.994492 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:33.994813 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:24:33.995291 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260929 16:24:33.998277 24137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } }
I20260929 16:24:33.998579 24137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:33.998656 24137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b52da3303c4b4089b6a12b6b92a593, State: Initialized, Role: FOLLOWER
I20260929 16:24:33.999148 24137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } }
I20260929 16:24:33.999294 24137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:33.999344 24137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:33.999418 24137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:34.000326 24137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } }
I20260929 16:24:34.000631 24137 leader_election.cc:302] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b52da3303c4b4089b6a12b6b92a593; no voters:
I20260929 16:24:34.000903 24137 leader_election.cc:288] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:34.001111 24142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:34.001407 24142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [term 1 LEADER]: Becoming Leader. State: Replica: 75b52da3303c4b4089b6a12b6b92a593, State: Running, Role: LEADER
I20260929 16:24:34.001747 24142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } }
I20260929 16:24:34.002223 24137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:34.003631 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b52da3303c4b4089b6a12b6b92a593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } } }
I20260929 16:24:34.003649 24144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b52da3303c4b4089b6a12b6b92a593. Latest consensus state: current_term: 1 leader_uuid: "75b52da3303c4b4089b6a12b6b92a593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b52da3303c4b4089b6a12b6b92a593" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 42941 } } }
I20260929 16:24:34.003796 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:34.003825 24144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:34.004176 24151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:34.007658 24151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:34.013783 24151 catalog_manager.cc:1347] Generated new cluster ID: 8cc1477597544caebdde998c1fd8392a
I20260929 16:24:34.013883 24151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:34.033159 24151 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:34.034183 24151 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:34.038442 24151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: Generated new TSK 0
I20260929 16:24:34.039198 24151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.200.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
Entry for principal kudu/127.4.200.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab.
I20260929 16:24:34.078161 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:34.193038 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:34.193305 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:34.193346 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:34.197471 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:34.197530 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:34.197634 24163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:34.203040 24163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.65.keytab
--principal=kudu/127.4.200.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:34.204209 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:34.205258 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:34.214941 24168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:34.214936 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:34.214941 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:34.215219 24163 server_base.cc:1034] running on GCE node
I20260929 16:24:34.215732 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:34.216358 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:34.217533 24163 hybrid_clock.cc:648] HybridClock initialized: now 1790699074217498 us; error 44 us; skew 500 ppm
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:34.220645 24163 init.cc:377] Logged in from keytab as kudu/127.4.200.65@KRBTEST.COM (short username kudu)
I20260929 16:24:34.221642 24163 webserver.cc:460] Webserver started at http://127.4.200.65:45671/ using document root <none> and password file <none>
I20260929 16:24:34.222251 24163 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:34.222355 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:34.222649 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:34.224835 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "5230955407884f358681e17e8ea32727"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "ba9bfc7925ce9ee5cd59112d3484bda9"
server_key_iv: "b9022a838a0f9b09c99799d76133d860"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.225466 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "5230955407884f358681e17e8ea32727"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "ba9bfc7925ce9ee5cd59112d3484bda9"
server_key_iv: "b9022a838a0f9b09c99799d76133d860"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.229496 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 16:24:34.232141 24178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.233362 24163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:34.233522 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "5230955407884f358681e17e8ea32727"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "ba9bfc7925ce9ee5cd59112d3484bda9"
server_key_iv: "b9022a838a0f9b09c99799d76133d860"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.233651 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:34.250730 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:34.253866 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:34.254079 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:34.254750 24163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:34.255581 24163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:34.255658 24163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.255739 24163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:34.255775 24163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.264735 24163 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:35423
I20260929 16:24:34.264752 24291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:35423 every 8 connection(s)
I20260929 16:24:34.265646 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.65@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:34.275076 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24163
I20260929 16:24:34.275231 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:34.275599 4897 external_mini_cluster.cc:1428] Setting key 90b1d6530fe4b4cfe7733b071eae9783
I20260929 16:24:34.279227 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.267435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:53937 (local address 127.4.200.126:42941)
0929 16:24:34.267607 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.267612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.268462 (+ 850us) server_negotiation.cc:407] Connection header received
0929 16:24:34.269285 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.269288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.269334 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.269410 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:34.270625 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.271151 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.272242 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.272459 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.272538 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.275093 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:34.275100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:34.275103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:34.275131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:34.276738 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.277361 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.277364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.277365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.277443 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.277841 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.277843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.277844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.278024 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:34.278101 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.278836 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.279006 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:24:34.280102 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.267835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42941 (local address 127.4.200.65:53937)
0929 16:24:34.268305 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.268348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.269092 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.269593 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.269599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.269943 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:34.270482 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.270505 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.271259 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.271261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.272138 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.272147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.272603 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.272606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.272739 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.273374 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:34.273391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:34.274909 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:34.276875 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.276881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.276900 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.277258 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.277542 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.277545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.277549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.277744 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.278129 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:34.278138 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:34.278512 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.278730 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.279118 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260929 16:24:34.281649 24292 heartbeater.cc:347] Connected to a master server at 127.4.200.126:42941
I20260929 16:24:34.281927 24292 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:34.282606 24292 heartbeater.cc:511] Master 127.4.200.126:42941 requested a full tablet report, sending...
I20260929 16:24:34.284515 24101 ts_manager.cc:194] Registered new tserver with Master: 5230955407884f358681e17e8ea32727 (127.4.200.65:35423)
WARNING: no policy specified for kudu/127.4.200.66@KRBTEST.COM; defaulting to no policy
I20260929 16:24:34.286213 24101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.65@KRBTEST.COM'} at 127.4.200.65:53937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
Entry for principal kudu/127.4.200.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab.
I20260929 16:24:34.305855 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:34.415269 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:34.415549 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:34.415620 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:34.420091 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:34.420166 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:34.420264 24297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:34.425582 24297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.66.keytab
--principal=kudu/127.4.200.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:34.426707 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:34.427700 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:34.437189 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:34.437196 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:34.437245 24297 server_base.cc:1034] running on GCE node
W20260929 16:24:34.437208 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:34.437857 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:34.438469 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:34.439644 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790699074439630 us; error 33 us; skew 500 ppm
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:34.442776 24297 init.cc:377] Logged in from keytab as kudu/127.4.200.66@KRBTEST.COM (short username kudu)
I20260929 16:24:34.443652 24297 webserver.cc:460] Webserver started at http://127.4.200.66:42651/ using document root <none> and password file <none>
I20260929 16:24:34.444237 24297 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:34.444299 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:34.444538 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:34.446823 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "88bf481c3e6a444b94020cfd3face492"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "290c2fbb03e20823df8a742bd46ac887"
server_key_iv: "b10d6465f67e70c66e5eb1d3335b3033"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.447402 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "88bf481c3e6a444b94020cfd3face492"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "290c2fbb03e20823df8a742bd46ac887"
server_key_iv: "b10d6465f67e70c66e5eb1d3335b3033"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.451148 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:34.453696 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.454984 24297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:34.455080 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "88bf481c3e6a444b94020cfd3face492"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "290c2fbb03e20823df8a742bd46ac887"
server_key_iv: "b10d6465f67e70c66e5eb1d3335b3033"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.455222 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:34.465006 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:34.468163 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:34.468384 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:34.468979 24297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:34.469813 24297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:34.469861 24297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.469929 24297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:34.469962 24297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.478916 24297 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:34955
I20260929 16:24:34.478937 24425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:34955 every 8 connection(s)
I20260929 16:24:34.479809 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:34.482368 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24297
I20260929 16:24:34.482566 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:34.482895 4897 external_mini_cluster.cc:1428] Setting key 0326059129c82209f5a05e01fe40e2ad
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.66@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.200.67@KRBTEST.COM; defaulting to no policy
I20260929 16:24:34.493938 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.481584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:37217 (local address 127.4.200.126:42941)
0929 16:24:34.481727 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.481731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.483024 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 16:24:34.484010 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.484013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.484067 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.484146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:34.485718 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.486230 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.487448 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.487689 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.487810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.490233 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:34.490239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:34.490241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:34.490270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:34.491797 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.492388 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.492391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.492393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.492475 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.492830 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.492835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.492836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.493024 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:34.493153 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.493652 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.493774 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:24:34.494877 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.481941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42941 (local address 127.4.200.66:37217)
0929 16:24:34.482856 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.482913 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.483796 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.484311 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.484317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.484845 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:34.485554 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.485576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.486376 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.486380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.487315 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.487325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.487846 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.487850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.488023 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.488648 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:34.488664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:34.490068 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:34.491929 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.491935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.491948 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.492291 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.492591 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.492594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.492596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.492731 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.493178 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:34.493184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:34.493414 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.493608 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.493930 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.67@KRBTEST.COM" created.
I20260929 16:24:34.496188 24426 heartbeater.cc:347] Connected to a master server at 127.4.200.126:42941
I20260929 16:24:34.496439 24426 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:34.496920 24426 heartbeater.cc:511] Master 127.4.200.126:42941 requested a full tablet report, sending...
I20260929 16:24:34.497961 24101 ts_manager.cc:194] Registered new tserver with Master: 88bf481c3e6a444b94020cfd3face492 (127.4.200.66:34955)
I20260929 16:24:34.498627 24101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.66@KRBTEST.COM'} at 127.4.200.66:37217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
Entry for principal kudu/127.4.200.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab.
I20260929 16:24:34.513923 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:34.620929 24431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:34.621153 24431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:34.621194 24431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:34.625029 24431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:34.625079 24431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:34.625133 24431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:34.629956 24431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.67.keytab
--principal=kudu/127.4.200.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:34.630868 24431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:34.631742 24431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:34.640599 24436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:34.640606 24437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:34.640815 24431 server_base.cc:1034] running on GCE node
W20260929 16:24:34.640614 24439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:34.641290 24431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:34.641858 24431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:34.643045 24431 hybrid_clock.cc:648] HybridClock initialized: now 1790699074643032 us; error 24 us; skew 500 ppm
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:34.645632 24431 init.cc:377] Logged in from keytab as kudu/127.4.200.67@KRBTEST.COM (short username kudu)
I20260929 16:24:34.646483 24431 webserver.cc:460] Webserver started at http://127.4.200.67:36971/ using document root <none> and password file <none>
I20260929 16:24:34.647051 24431 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:34.647142 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:34.647368 24431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:34.649300 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "2a171f19246f44a89439f6da83027ca1"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "2fc5c0a2a3ae41534d9c011599944618"
server_key_iv: "bab4fa51932aad790bd83aee84e66dc6"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.649845 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "2a171f19246f44a89439f6da83027ca1"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "2fc5c0a2a3ae41534d9c011599944618"
server_key_iv: "bab4fa51932aad790bd83aee84e66dc6"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.653636 24431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:34.656201 24446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.657404 24431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:34.657543 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "2a171f19246f44a89439f6da83027ca1"
format_stamp: "Formatted at 2026-09-29 16:24:34 on dist-test-slave-s9ct"
server_key: "2fc5c0a2a3ae41534d9c011599944618"
server_key_iv: "bab4fa51932aad790bd83aee84e66dc6"
server_key_version: "encryptionkey@0"
I20260929 16:24:34.657655 24431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:34.673987 24431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:34.676767 24431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:34.676975 24431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:34.677559 24431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:34.678366 24431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:34.678484 24431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.678574 24431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:34.678628 24431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:34.687778 24431 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:33123
I20260929 16:24:34.687803 24559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:33123 every 8 connection(s)
I20260929 16:24:34.688634 24431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:34.689823 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24431
I20260929 16:24:34.689983 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:34.690284 4897 external_mini_cluster.cc:1428] Setting key 05efea8889846b7967b62b3fb3be6c32
Sep 29 16:24:34 dist-test-slave-s9ct krb5kdc[24056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.67@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:34.701808 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.690477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:39475 (local address 127.4.200.126:42941)
0929 16:24:34.690651 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.690656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.691582 (+ 926us) server_negotiation.cc:407] Connection header received
0929 16:24:34.692738 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.692741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.692798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.692869 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:34.694010 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.694616 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.695763 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.695987 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.696102 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.698447 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:34.698451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:34.698453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:34.698476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:34.699753 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.700258 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.700263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.700265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.700346 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.700689 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.700692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.700694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.700875 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:34.700990 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.701468 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.701591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:24:34.702725 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.690942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:42941 (local address 127.4.200.67:39475)
0929 16:24:34.691421 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.691477 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.692494 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.693066 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.693074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.693418 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:34.693867 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.693879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.694749 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.694753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.695607 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.695616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.696133 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.696137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.696317 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.696838 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:24:34.696853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:34.698253 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:34.699874 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.699880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.699893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.700143 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.700442 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:24:34.700447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:24:34.700449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:24:34.700585 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:24:34.700986 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:34.700993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:34.701238 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.701429 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.701741 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260929 16:24:34.703881 24560 heartbeater.cc:347] Connected to a master server at 127.4.200.126:42941
I20260929 16:24:34.704142 24560 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:34.704669 24560 heartbeater.cc:511] Master 127.4.200.126:42941 requested a full tablet report, sending...
I20260929 16:24:34.705667 24100 ts_manager.cc:194] Registered new tserver with Master: 2a171f19246f44a89439f6da83027ca1 (127.4.200.67:33123)
I20260929 16:24:34.706264 24100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.200.67@KRBTEST.COM'} at 127.4.200.67:39475
I20260929 16:24:34.716132 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:34.725776 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.718199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.4.200.126:42941)
0929 16:24:34.718329 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.718333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.718469 (+ 136us) server_negotiation.cc:407] Connection header received
0929 16:24:34.718614 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.718617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.718658 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.718751 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:34.719451 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.719966 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.720826 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.721045 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.721159 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.722550 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:34.722553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:34.722555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:34.722578 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:34.724032 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.724473 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.724477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.724479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.724560 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:34.724847 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:34.724852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:34.724854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:34.725082 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:34.725228 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.725480 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.725595 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:24:34.728783 24100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:34.731899 24100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:34.745523 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.740004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:34955 (local address 127.0.0.1:40782)
0929 16:24:34.740345 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.740388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.740502 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.741090 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.741095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.741119 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:34.741277 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.741285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.742912 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.742918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.744151 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.744160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.744992 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.744996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.745112 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.745148 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.745270 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.745384 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
I20260929 16:24:34.745535 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.740148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.4.200.66:34955)
0929 16:24:34.740446 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.740456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.740478 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:24:34.740696 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.740703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.740911 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.741432 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:34.741614 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.742731 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.744308 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.744864 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.744995 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.745142 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.745299 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.745387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260929 16:24:34.747061 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.740776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33123 (local address 127.0.0.1:58818)
0929 16:24:34.740989 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.741005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.741084 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.742356 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.742359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.742377 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:34.742505 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.742515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.744052 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.744056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.745543 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.745554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.746659 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.746663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.746758 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.746779 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.746832 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.746898 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":110,"thread_start_us":94,"threads_started":1}
I20260929 16:24:34.747336 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.741430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.4.200.65:35423)
0929 16:24:34.741718 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.741732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.741986 (+ 254us) server_negotiation.cc:407] Connection header received
0929 16:24:34.742078 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.742083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.742224 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.742364 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:34.742838 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.743956 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.745693 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.746521 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.746636 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.746782 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.747076 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.747174 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":133,"threads_started":1}
I20260929 16:24:34.747664 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.741099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:35423 (local address 127.0.0.1:33076)
0929 16:24:34.741872 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.741886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.741984 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.742620 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.742622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.742635 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:34.742717 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.742721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.744086 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.744089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.745543 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.745554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.746842 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.746845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.746929 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.746948 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.747469 (+ 521us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.747547 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":168,"thread_start_us":85,"threads_started":1}
I20260929 16:24:34.747754 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.741479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.4.200.67:33123)
0929 16:24:34.741891 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.741899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.741918 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:24:34.742015 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.742022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.742204 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.742364 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:34.742605 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.743898 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.745683 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.746521 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.747343 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.747490 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.747536 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.747616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260929 16:24:34.749020 24360 tablet_service.cc:1467] Processing CreateTablet for tablet 41b9a9354355475c9a4b445fde335908 (DEFAULT_TABLE table=test-table [id=f91c87b057ad410e86622fbb3bba0c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:34.749907 24360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b9a9354355475c9a4b445fde335908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:34.750355 24494 tablet_service.cc:1467] Processing CreateTablet for tablet 41b9a9354355475c9a4b445fde335908 (DEFAULT_TABLE table=test-table [id=f91c87b057ad410e86622fbb3bba0c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:34.750355 24226 tablet_service.cc:1467] Processing CreateTablet for tablet 41b9a9354355475c9a4b445fde335908 (DEFAULT_TABLE table=test-table [id=f91c87b057ad410e86622fbb3bba0c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:34.751377 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b9a9354355475c9a4b445fde335908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:34.751371 24226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b9a9354355475c9a4b445fde335908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:34.756775 24577 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Bootstrap starting.
I20260929 16:24:34.758644 24578 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Bootstrap starting.
I20260929 16:24:34.759481 24577 tablet_bootstrap.cc:654] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:34.760211 24577 log.cc:824] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:34.760838 24578 tablet_bootstrap.cc:654] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:34.761387 24579 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: Bootstrap starting.
I20260929 16:24:34.761654 24578 log.cc:824] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:34.762447 24577 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: No bootstrap required, opened a new log
I20260929 16:24:34.762641 24577 ts_tablet_manager.cc:1397] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:34.763329 24579 tablet_bootstrap.cc:654] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:34.763607 24578 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: No bootstrap required, opened a new log
I20260929 16:24:34.763782 24578 ts_tablet_manager.cc:1397] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:34.764340 24579 log.cc:824] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:34.765972 24579 tablet_bootstrap.cc:492] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: No bootstrap required, opened a new log
I20260929 16:24:34.766168 24579 ts_tablet_manager.cc:1397] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:34.766607 24577 raft_consensus.cc:348] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.766896 24577 raft_consensus.cc:374] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:34.766958 24577 raft_consensus.cc:729] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bf481c3e6a444b94020cfd3face492, State: Initialized, Role: FOLLOWER
I20260929 16:24:34.767388 24577 consensus_queue.cc:260] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.767590 24578 raft_consensus.cc:348] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.767831 24578 raft_consensus.cc:374] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:34.767877 24578 raft_consensus.cc:729] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a171f19246f44a89439f6da83027ca1, State: Initialized, Role: FOLLOWER
I20260929 16:24:34.768366 24577 ts_tablet_manager.cc:1428] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:34.768296 24578 consensus_queue.cc:260] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.768709 24426 heartbeater.cc:503] Master 127.4.200.126:42941 was elected leader, sending a full tablet report...
I20260929 16:24:34.769591 24579 raft_consensus.cc:348] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.769833 24579 raft_consensus.cc:374] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:34.769888 24579 raft_consensus.cc:729] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5230955407884f358681e17e8ea32727, State: Initialized, Role: FOLLOWER
I20260929 16:24:34.769965 24578 ts_tablet_manager.cc:1428] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:24:34.770304 24560 heartbeater.cc:503] Master 127.4.200.126:42941 was elected leader, sending a full tablet report...
I20260929 16:24:34.770305 24579 consensus_queue.cc:260] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.770911 24292 heartbeater.cc:503] Master 127.4.200.126:42941 was elected leader, sending a full tablet report...
I20260929 16:24:34.771260 24579 ts_tablet_manager.cc:1428] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:24:34.807413 24585 raft_consensus.cc:482] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:34.807612 24585 raft_consensus.cc:504] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.808738 24585 leader_election.cc:288] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88bf481c3e6a444b94020cfd3face492 (127.4.200.66:34955), 2a171f19246f44a89439f6da83027ca1 (127.4.200.67:33123)
I20260929 16:24:34.813135 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.809157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:55579 (local address 127.4.200.66:34955)
0929 16:24:34.809323 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.809330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.809583 (+ 253us) server_negotiation.cc:407] Connection header received
0929 16:24:34.809910 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.809915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.810021 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.810088 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:34.810568 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.811064 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812192 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.812594 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812726 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.812781 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.812947 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.813012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:24:34.813164 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.809003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:34955 (local address 127.4.200.65:55579)
0929 16:24:34.809430 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.809450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.809670 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.810229 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.810231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.810245 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:34.810380 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.810385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.811180 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.811187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.812058 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.812067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.812714 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.812817 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.812842 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.812933 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.813003 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":50,"thread_start_us":110,"threads_started":1}
I20260929 16:24:34.813254 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.809797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:45127 (local address 127.4.200.67:33123)
0929 16:24:34.809911 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.809914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.809926 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:24:34.810133 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.810136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.810186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.810272 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:34.810603 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.811097 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812289 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.812715 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812847 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.812895 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.813065 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.813131 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:24:34.813302 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.809415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:33123 (local address 127.4.200.65:45127)
0929 16:24:34.809729 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.809743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:34.809991 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:34.810283 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:34.810285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:34.810301 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:34.810489 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.810496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.811216 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.811221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.812162 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:34.812172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.812839 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.812842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:34.812936 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.812963 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:24:34.813051 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:24:34.813118 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":59,"thread_start_us":124,"threads_started":1}
I20260929 16:24:34.813869 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b9a9354355475c9a4b445fde335908" candidate_uuid: "5230955407884f358681e17e8ea32727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a171f19246f44a89439f6da83027ca1" is_pre_election: true
I20260929 16:24:34.813975 24380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b9a9354355475c9a4b445fde335908" candidate_uuid: "5230955407884f358681e17e8ea32727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf481c3e6a444b94020cfd3face492" is_pre_election: true
I20260929 16:24:34.814213 24514 raft_consensus.cc:2454] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5230955407884f358681e17e8ea32727 in term 0.
I20260929 16:24:34.814288 24380 raft_consensus.cc:2454] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5230955407884f358681e17e8ea32727 in term 0.
I20260929 16:24:34.814796 24180 leader_election.cc:302] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a171f19246f44a89439f6da83027ca1, 5230955407884f358681e17e8ea32727; no voters:
I20260929 16:24:34.815088 24585 raft_consensus.cc:2781] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:34.815212 24585 raft_consensus.cc:482] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:34.815270 24585 raft_consensus.cc:3037] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:34.816416 24585 raft_consensus.cc:504] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.816776 24585 leader_election.cc:288] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [CANDIDATE]: Term 1 election: Requested vote from peers 88bf481c3e6a444b94020cfd3face492 (127.4.200.66:34955), 2a171f19246f44a89439f6da83027ca1 (127.4.200.67:33123)
I20260929 16:24:34.817168 24380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b9a9354355475c9a4b445fde335908" candidate_uuid: "5230955407884f358681e17e8ea32727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf481c3e6a444b94020cfd3face492"
I20260929 16:24:34.817241 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b9a9354355475c9a4b445fde335908" candidate_uuid: "5230955407884f358681e17e8ea32727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a171f19246f44a89439f6da83027ca1"
I20260929 16:24:34.817296 24380 raft_consensus.cc:3037] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:34.817391 24514 raft_consensus.cc:3037] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:34.818437 24380 raft_consensus.cc:2454] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5230955407884f358681e17e8ea32727 in term 1.
I20260929 16:24:34.818482 24514 raft_consensus.cc:2454] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5230955407884f358681e17e8ea32727 in term 1.
I20260929 16:24:34.818737 24180 leader_election.cc:302] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a171f19246f44a89439f6da83027ca1, 5230955407884f358681e17e8ea32727; no voters:
I20260929 16:24:34.818943 24585 raft_consensus.cc:2781] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:34.819168 24585 raft_consensus.cc:686] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 1 LEADER]: Becoming Leader. State: Replica: 5230955407884f358681e17e8ea32727, State: Running, Role: LEADER
I20260929 16:24:34.819496 24585 consensus_queue.cc:237] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } }
I20260929 16:24:34.822741 24100 catalog_manager.cc:5495] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5230955407884f358681e17e8ea32727 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "5230955407884f358681e17e8ea32727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5230955407884f358681e17e8ea32727" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260929 16:24:34.844014 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.840227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.4.200.65:35423)
0929 16:24:34.840423 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:34.840428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:34.840442 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:24:34.840554 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:34.840556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:34.840639 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:34.840817 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:34.841073 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.841583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.842508 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:34.842730 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:34.842850 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:34.843171 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:34.843602 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:34.843747 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:34.843791 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 16:24:34.843854 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:24:34.849925 24380 raft_consensus.cc:1261] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 1 FOLLOWER]: Refusing update from remote peer 5230955407884f358681e17e8ea32727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:34.849925 24514 raft_consensus.cc:1261] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 1 FOLLOWER]: Refusing update from remote peer 5230955407884f358681e17e8ea32727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:34.850672 24585 consensus_queue.cc:1035] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88bf481c3e6a444b94020cfd3face492" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:34.850927 24588 consensus_queue.cc:1035] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a171f19246f44a89439f6da83027ca1" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:34.857306 24591 mvcc.cc:205] Tried to move back new op lower bound from 7334703410577993728 to 7334703410463485952. Current Snapshot: MvccSnapshot[applied={T|T < 7334703410577993728}]
I20260929 16:24:34.858291 24593 mvcc.cc:205] Tried to move back new op lower bound from 7334703410577993728 to 7334703410463485952. Current Snapshot: MvccSnapshot[applied={T|T < 7334703410577993728}]
I20260929 16:24:34.861738 24592 mvcc.cc:205] Tried to move back new op lower bound from 7334703410577993728 to 7334703410463485952. Current Snapshot: MvccSnapshot[applied={T|T < 7334703410577993728 or (T in {7334703410577993728})}]
I20260929 16:24:34.866765 24101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:34.866911 24101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:34.869385 24101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75b52da3303c4b4089b6a12b6b92a593: Sending DeleteTablet for 3 replicas of tablet 41b9a9354355475c9a4b445fde335908
I20260929 16:24:34.870172 24360 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b9a9354355475c9a4b445fde335908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:34 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:40782
I20260929 16:24:34.870239 24494 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b9a9354355475c9a4b445fde335908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:34 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:58818
I20260929 16:24:34.870646 24226 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b9a9354355475c9a4b445fde335908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:34 UTC) from {username='kudu', principal='kudu/127.4.200.126@KRBTEST.COM'} at 127.0.0.1:33076
I20260929 16:24:34.870780 24600 tablet_replica.cc:330] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: stopping tablet replica
I20260929 16:24:34.871099 24600 raft_consensus.cc:2229] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:34.871135 24602 tablet_replica.cc:330] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727: stopping tablet replica
I20260929 16:24:34.871431 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24163
I20260929 16:24:34.871428 24600 raft_consensus.cc:2258] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:34.871630 24602 raft_consensus.cc:2229] T 41b9a9354355475c9a4b445fde335908 P 5230955407884f358681e17e8ea32727 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:24:34.872313 24601 tablet_replica.cc:330] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: stopping tablet replica
I20260929 16:24:34.872607 24601 raft_consensus.cc:2229] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:34.872890 24601 raft_consensus.cc:2258] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:34.874454 24600 ts_tablet_manager.cc:1905] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:34.875555 24601 ts_tablet_manager.cc:1905] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:34.877790 24601 ts_tablet_manager.cc:1918] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:34.877911 24601 log.cc:1196] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/41b9a9354355475c9a4b445fde335908
I20260929 16:24:34.877908 24600 ts_tablet_manager.cc:1918] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:34.877996 24600 log.cc:1196] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/41b9a9354355475c9a4b445fde335908
I20260929 16:24:34.878340 24600 ts_tablet_manager.cc:1939] T 41b9a9354355475c9a4b445fde335908 P 88bf481c3e6a444b94020cfd3face492: Deleting consensus metadata
I20260929 16:24:34.878341 24601 ts_tablet_manager.cc:1939] T 41b9a9354355475c9a4b445fde335908 P 2a171f19246f44a89439f6da83027ca1: Deleting consensus metadata
I20260929 16:24:34.879581 24086 catalog_manager.cc:4841] TS 88bf481c3e6a444b94020cfd3face492 (127.4.200.66:34955): tablet 41b9a9354355475c9a4b445fde335908 (table test-table [id=f91c87b057ad410e86622fbb3bba0c02]) successfully deleted
I20260929 16:24:34.879832 24086 catalog_manager.cc:4841] TS 2a171f19246f44a89439f6da83027ca1 (127.4.200.67:33123): tablet 41b9a9354355475c9a4b445fde335908 (table test-table [id=f91c87b057ad410e86622fbb3bba0c02]) successfully deleted
W20260929 16:24:34.880036 24086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:35423 (error 108)
I20260929 16:24:34.880075 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24297
I20260929 16:24:34.881525 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:34.881163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:35423 (local address 127.0.0.1:33090)
0929 16:24:34.881330 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:34.881419 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:35423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:24:34.881703 24086 catalog_manager.cc:4584] TS 5230955407884f358681e17e8ea32727 (127.4.200.65:35423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41b9a9354355475c9a4b445fde335908: Network error: Client connection negotiation failed: client connection to 127.4.200.65:35423: connect: Connection refused (error 111)
I20260929 16:24:34.886534 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24431
I20260929 16:24:34.892220 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24069
2026-09-29T16:24:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2271 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:34Z Disabled control of system clock
I20260929 16:24:34.923708 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40525
--webserver_interface=127.4.200.126
--webserver_port=0
--builtin_ntp_servers=127.4.200.84:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:35.035617 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:35.035907 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:35.035943 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:35.039889 24607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:35.039944 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:35.039960 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:35.039978 24607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:35.039992 24607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:35.044637 24607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:40525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:35.045678 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:35.047147 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:35.055616 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.055720 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.055616 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:35.055919 24607 server_base.cc:1034] running on GCE node
I20260929 16:24:35.056582 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:35.057746 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:35.058952 24607 hybrid_clock.cc:648] HybridClock initialized: now 1790699075058921 us; error 44 us; skew 500 ppm
I20260929 16:24:35.060916 24607 webserver.cc:460] Webserver started at http://127.4.200.126:41405/ using document root <none> and password file <none>
I20260929 16:24:35.061519 24607 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:35.061616 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:35.061831 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:35.063987 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "190eaa8174ee4516a71e7eea0b65c8e2"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "68991a2e5eab07014c51bdb5334f6925"
server_key_iv: "b81fda18c04ee7e036afb65ed3b021a5"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.064574 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "190eaa8174ee4516a71e7eea0b65c8e2"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "68991a2e5eab07014c51bdb5334f6925"
server_key_iv: "b81fda18c04ee7e036afb65ed3b021a5"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.068490 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:24:35.071004 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.072252 24607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:35.072393 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "190eaa8174ee4516a71e7eea0b65c8e2"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "68991a2e5eab07014c51bdb5334f6925"
server_key_iv: "b81fda18c04ee7e036afb65ed3b021a5"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.072510 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:35.088876 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:35.089651 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:35.089870 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:35.096776 24607 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:40525
I20260929 16:24:35.096783 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:40525 every 8 connection(s)
I20260929 16:24:35.097765 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:35.099619 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24607
I20260929 16:24:35.099777 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:35.100135 4897 external_mini_cluster.cc:1428] Setting key 42b3300474812d2b667b979f1965430f
I20260929 16:24:35.101740 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:35.108108 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: Bootstrap starting.
I20260929 16:24:35.109959 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.101712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.4.200.126:40525)
0929 16:24:35.102374 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.102448 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.102493 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:24:35.103150 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.103166 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.103572 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.103848 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:35.104211 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.105161 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.106370 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.106970 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.107101 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.108006 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:35.108012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:35.108026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:35.108482 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:35.108536 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:35.108558 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:35.108658 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.108882 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.109202 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260929 16:24:35.110546 24674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:35.111272 24674 log.cc:824] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:35.113204 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: No bootstrap required, opened a new log
I20260929 16:24:35.116685 24674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } }
I20260929 16:24:35.116940 24674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:35.117019 24674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190eaa8174ee4516a71e7eea0b65c8e2, State: Initialized, Role: FOLLOWER
I20260929 16:24:35.117443 24674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } }
I20260929 16:24:35.117590 24674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:35.117674 24674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:35.117794 24674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:35.118819 24674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } }
I20260929 16:24:35.119160 24674 leader_election.cc:302] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190eaa8174ee4516a71e7eea0b65c8e2; no voters:
I20260929 16:24:35.119463 24674 leader_election.cc:288] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:35.119601 24679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:35.119874 24679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 190eaa8174ee4516a71e7eea0b65c8e2, State: Running, Role: LEADER
I20260929 16:24:35.120195 24679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } }
I20260929 16:24:35.120716 24674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:35.121850 24680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } } }
I20260929 16:24:35.121874 24681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190eaa8174ee4516a71e7eea0b65c8e2. Latest consensus state: current_term: 1 leader_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190eaa8174ee4516a71e7eea0b65c8e2" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40525 } } }
I20260929 16:24:35.121989 24681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:35.121989 24680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:35.122516 24688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:35.125635 24688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:35.139853 24688 catalog_manager.cc:1347] Generated new cluster ID: 385d31b8a26c4e23abef41c91c4ce0cb
I20260929 16:24:35.139945 24688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:35.150342 24688 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:35.151353 24688 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:35.156993 24688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: Generated new TSK 0
I20260929 16:24:35.157732 24688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:35.160410 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.65:0
--local_ip_for_outbound_sockets=127.4.200.65
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--builtin_ntp_servers=127.4.200.84:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:24:35.271785 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:35.272059 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:35.272101 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:35.276297 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:35.276358 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:35.276446 24698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.65
I20260929 16:24:35.281276 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.200.65
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.65
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:35.282447 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:35.283874 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:35.293372 24706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.293409 24703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.293414 24704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:35.293606 24698 server_base.cc:1034] running on GCE node
I20260929 16:24:35.294137 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:35.294888 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:35.296037 24698 hybrid_clock.cc:648] HybridClock initialized: now 1790699075296021 us; error 27 us; skew 500 ppm
I20260929 16:24:35.298038 24698 webserver.cc:460] Webserver started at http://127.4.200.65:35629/ using document root <none> and password file <none>
I20260929 16:24:35.298714 24698 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:35.298780 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:35.298992 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:35.301075 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data/instance:
uuid: "4bf0462b89c94ccaa3cd4a895d1497a3"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "8e5f08dcbf0e3301e9bdf577a60d27c2"
server_key_iv: "170191cbd2237ba9449a907d1902d001"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.301651 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance:
uuid: "4bf0462b89c94ccaa3cd4a895d1497a3"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "8e5f08dcbf0e3301e9bdf577a60d27c2"
server_key_iv: "170191cbd2237ba9449a907d1902d001"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.305439 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:35.308090 24712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.309388 24698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:35.309558 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal
uuid: "4bf0462b89c94ccaa3cd4a895d1497a3"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "8e5f08dcbf0e3301e9bdf577a60d27c2"
server_key_iv: "170191cbd2237ba9449a907d1902d001"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.309700 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:35.332242 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:35.333014 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:35.333228 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:35.333854 24698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:35.334869 24698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:35.334954 24698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.335022 24698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:35.335055 24698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.344430 24698 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.65:35951
I20260929 16:24:35.344450 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.65:35951 every 8 connection(s)
I20260929 16:24:35.345381 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/data/info.pb
I20260929 16:24:35.347323 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24698
I20260929 16:24:35.347465 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-0/wal/instance
I20260929 16:24:35.347769 4897 external_mini_cluster.cc:1428] Setting key a47522f69524192bc397df5d8c270de8
I20260929 16:24:35.350104 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.66:0
--local_ip_for_outbound_sockets=127.4.200.66
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--builtin_ntp_servers=127.4.200.84:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:35.355665 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.347310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:57665 (local address 127.4.200.126:40525)
0929 16:24:35.347483 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.347489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.348667 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 16:24:35.350492 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.350497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.350563 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.350643 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:35.351857 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.352411 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.353576 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.353820 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.353939 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.355130 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:35.355133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:35.355141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:35.355154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:35.355175 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:35.355189 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:35.355268 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.355432 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.355515 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:24:35.356570 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.347740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:40525 (local address 127.4.200.65:57665)
0929 16:24:35.348501 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.348558 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.350242 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.350833 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.350844 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.350963 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:35.351700 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.351726 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.352553 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.352556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.353443 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.353452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.353981 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.353984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.354155 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.354861 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:35.354863 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:35.354904 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:35.354970 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:35.355282 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:35.355292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:35.355303 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.355432 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.355756 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":117,"threads_started":1}
I20260929 16:24:35.357914 24826 heartbeater.cc:347] Connected to a master server at 127.4.200.126:40525
I20260929 16:24:35.358171 24826 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:35.358786 24826 heartbeater.cc:511] Master 127.4.200.126:40525 requested a full tablet report, sending...
I20260929 16:24:35.360540 24638 ts_manager.cc:194] Registered new tserver with Master: 4bf0462b89c94ccaa3cd4a895d1497a3 (127.4.200.65:35951)
I20260929 16:24:35.362164 24638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.65:57665
W20260929 16:24:35.464809 24829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:35.465049 24829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:35.465086 24829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:35.469246 24829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:35.469301 24829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:35.469357 24829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.66
I20260929 16:24:35.473989 24829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.200.66
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.66
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:35.474938 24829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:35.476341 24829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:35.485601 24834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.485642 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.485601 24835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:35.485837 24829 server_base.cc:1034] running on GCE node
I20260929 16:24:35.486487 24829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:35.487138 24829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:35.488319 24829 hybrid_clock.cc:648] HybridClock initialized: now 1790699075488299 us; error 32 us; skew 500 ppm
I20260929 16:24:35.490108 24829 webserver.cc:460] Webserver started at http://127.4.200.66:45501/ using document root <none> and password file <none>
I20260929 16:24:35.490749 24829 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:35.490844 24829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:35.491078 24829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:35.493068 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data/instance:
uuid: "da65efd14430469287c315e8fb7a5790"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "1565339dfbee32ed5b2fce7416049e50"
server_key_iv: "3f4b0f8c3b649cac39bd04ee44273693"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.493654 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance:
uuid: "da65efd14430469287c315e8fb7a5790"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "1565339dfbee32ed5b2fce7416049e50"
server_key_iv: "3f4b0f8c3b649cac39bd04ee44273693"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.497493 24829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:35.500180 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.501479 24829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:35.501634 24829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal
uuid: "da65efd14430469287c315e8fb7a5790"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "1565339dfbee32ed5b2fce7416049e50"
server_key_iv: "3f4b0f8c3b649cac39bd04ee44273693"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.501754 24829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:35.516727 24829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:35.517371 24829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:35.517721 24829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:35.518347 24829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:35.519307 24829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:35.519385 24829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.519452 24829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:35.519505 24829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.528661 24829 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.66:46721
I20260929 16:24:35.528688 24956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.66:46721 every 8 connection(s)
I20260929 16:24:35.529526 24829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/data/info.pb
I20260929 16:24:35.536924 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24829
I20260929 16:24:35.537068 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal/instance
I20260929 16:24:35.537482 4897 external_mini_cluster.cc:1428] Setting key 3f4f19b7d1c418c77105e45e3c2eb47a
I20260929 16:24:35.539630 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.200.67:0
--local_ip_for_outbound_sockets=127.4.200.67
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--builtin_ntp_servers=127.4.200.84:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:24:35.547989 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.531351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.66:42519 (local address 127.4.200.126:40525)
0929 16:24:35.531502 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.531508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.532542 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 16:24:35.533873 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.533876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.533926 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.534002 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:35.534887 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.535404 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.536493 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.536701 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.536779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.547376 (+ 10597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:35.547379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:35.547382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:35.547397 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:35.547418 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:35.547428 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:35.547519 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.547740 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.547832 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:24:35.548995 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.531782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:40525 (local address 127.4.200.66:42519)
0929 16:24:35.532389 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.532429 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.533660 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.534163 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.534169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.534253 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:35.534756 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.534768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.535509 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.535511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.536380 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.536389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.536874 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.536878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.537095 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.546911 (+ 9816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:35.546913 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:35.546970 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:35.547141 (+ 171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:35.547563 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:35.547580 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:35.547586 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.547740 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.548099 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260929 16:24:35.550387 24957 heartbeater.cc:347] Connected to a master server at 127.4.200.126:40525
I20260929 16:24:35.550753 24957 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:35.551304 24957 heartbeater.cc:511] Master 127.4.200.126:40525 requested a full tablet report, sending...
I20260929 16:24:35.552417 24638 ts_manager.cc:194] Registered new tserver with Master: da65efd14430469287c315e8fb7a5790 (127.4.200.66:46721)
I20260929 16:24:35.553191 24638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.66:42519
W20260929 16:24:35.655232 24960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:35.655515 24960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:35.655555 24960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:35.659966 24960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:35.660039 24960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:35.660099 24960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.200.67
I20260929 16:24:35.665011 24960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.200.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.200.67
--webserver_port=0
--tserver_master_addrs=127.4.200.126:40525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.200.67
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:35.666201 24960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:35.667758 24960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:35.677760 24965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.677769 24968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:35.677767 24966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:35.678150 24960 server_base.cc:1034] running on GCE node
I20260929 16:24:35.678642 24960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:35.679280 24960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:35.680459 24960 hybrid_clock.cc:648] HybridClock initialized: now 1790699075680442 us; error 32 us; skew 500 ppm
I20260929 16:24:35.682643 24960 webserver.cc:460] Webserver started at http://127.4.200.67:36717/ using document root <none> and password file <none>
I20260929 16:24:35.683238 24960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:35.683300 24960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:35.683511 24960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:35.685636 24960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data/instance:
uuid: "bb5d954b5474445aac48938be96d1b6e"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "3fe2658bad3621266e555ae3e04e1275"
server_key_iv: "dda35f59b5c436c6f2f18d37cf19d4d8"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.686224 24960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance:
uuid: "bb5d954b5474445aac48938be96d1b6e"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "3fe2658bad3621266e555ae3e04e1275"
server_key_iv: "dda35f59b5c436c6f2f18d37cf19d4d8"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.689986 24960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:35.692605 24974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.693987 24960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:35.694170 24960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal
uuid: "bb5d954b5474445aac48938be96d1b6e"
format_stamp: "Formatted at 2026-09-29 16:24:35 on dist-test-slave-s9ct"
server_key: "3fe2658bad3621266e555ae3e04e1275"
server_key_iv: "dda35f59b5c436c6f2f18d37cf19d4d8"
server_key_version: "encryptionkey@0"
I20260929 16:24:35.694316 24960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:35.711328 24960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:35.712116 24960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:35.712343 24960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:35.712972 24960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:24:35.714061 24960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:24:35.714148 24960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.714221 24960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:24:35.714274 24960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:35.723640 24960 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.67:44927
I20260929 16:24:35.723668 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.67:44927 every 8 connection(s)
I20260929 16:24:35.724561 24960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/data/info.pb
I20260929 16:24:35.726553 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 24960
I20260929 16:24:35.726699 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal/instance
I20260929 16:24:35.726998 4897 external_mini_cluster.cc:1428] Setting key 15c84fa1871c0b0c447f70c9ca64385f
I20260929 16:24:35.734859 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.726414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.67:59595 (local address 127.4.200.126:40525)
0929 16:24:35.726557 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.726563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.727841 (+ 1278us) server_negotiation.cc:407] Connection header received
0929 16:24:35.729835 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.729837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.729886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.729961 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:35.731047 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.731584 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.732838 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.733056 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.733140 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.734330 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:35.734332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:35.734334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:35.734347 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:35.734366 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:35.734374 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:35.734497 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.734673 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.734733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:24:35.735724 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.726933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:40525 (local address 127.4.200.67:59595)
0929 16:24:35.727689 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.727747 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.729603 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.730138 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.730145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.730237 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:24:35.730885 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.730901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.731714 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.731716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.732692 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.732702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.733226 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.733229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.733390 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.734035 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:24:35.734037 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:24:35.734078 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:24:35.734159 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:24:35.734514 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:24:35.734523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:24:35.734528 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.734649 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.734885 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260929 16:24:35.736920 25088 heartbeater.cc:347] Connected to a master server at 127.4.200.126:40525
I20260929 16:24:35.737203 25088 heartbeater.cc:464] Registering TS with master...
I20260929 16:24:35.737754 25088 heartbeater.cc:511] Master 127.4.200.126:40525 requested a full tablet report, sending...
I20260929 16:24:35.738912 24638 ts_manager.cc:194] Registered new tserver with Master: bb5d954b5474445aac48938be96d1b6e (127.4.200.67:44927)
I20260929 16:24:35.739503 24638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.200.67:59595
I20260929 16:24:35.741029 4897 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:24:35.746634 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.743140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.4.200.126:40525)
0929 16:24:35.743293 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.743299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.743417 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:24:35.743516 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.743518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.743556 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.743620 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:35.743836 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.744314 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.745093 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.745306 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.745380 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.746168 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:35.746171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:35.746173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:35.746184 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:35.746202 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:35.746209 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:35.746265 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.746376 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.746470 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:24:35.749282 24638 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:24:35.752346 24638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:24:35.767221 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.760596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:35951 (local address 127.0.0.1:48590)
0929 16:24:35.760978 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.761032 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.761166 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.762078 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.762084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.762110 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:35.762278 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.762287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.763877 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.763880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.765317 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.765326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.766572 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.766577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.766698 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.766757 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.766892 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.767035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":36,"thread_start_us":106,"threads_started":1}
I20260929 16:24:35.767278 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.760731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.4.200.65:35951)
0929 16:24:35.761590 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.761605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.761629 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:24:35.761704 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.761708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.761903 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.762464 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:35.762611 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.763749 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.765458 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.766374 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.766543 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.766759 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.766936 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.767082 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":134,"threads_started":1}
I20260929 16:24:35.767544 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.760851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.4.200.67:44927)
0929 16:24:35.761226 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.761241 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.761524 (+ 283us) server_negotiation.cc:407] Connection header received
0929 16:24:35.761673 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.761681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.761901 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.762064 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:35.762540 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.763748 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.765627 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.766446 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.766559 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.766759 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.767132 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.767381 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":164,"threads_started":1}
I20260929 16:24:35.768110 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.760596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:44927 (local address 127.0.0.1:48750)
0929 16:24:35.761403 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.761415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.761517 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.762250 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.762254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.762273 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:35.762388 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.762419 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.763905 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.763908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.765498 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.765509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.766829 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.766833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.766962 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.766987 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.767854 (+ 867us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.767936 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":744,"thread_start_us":104,"threads_started":1}
I20260929 16:24:35.768529 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.761574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46721 (local address 127.0.0.1:44368)
0929 16:24:35.762212 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:35.762225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:35.762303 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:35.762895 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:35.762899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:35.762917 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:35.763002 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.763010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.764150 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.764153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.765504 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:35.765510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.768122 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.768125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:35.768231 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.768254 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:24:35.768341 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:24:35.768410 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":89,"thread_start_us":90,"threads_started":1}
I20260929 16:24:35.768685 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:35.761993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.4.200.66:46721)
0929 16:24:35.762504 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:35.762519 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:35.762545 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:24:35.762619 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:35.762622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:35.762757 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:35.762919 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:35.763135 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.763990 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.766575 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:35.767632 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:35.767739 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:35.768049 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:35.768384 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:24:35.768511 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":149,"threads_started":1}
I20260929 16:24:35.771070 24760 tablet_service.cc:1467] Processing CreateTablet for tablet d34261daf02f459481b8998e480e4f76 (DEFAULT_TABLE table=test-table [id=0158ef67a9e14641bdf90bf8498e3a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:35.771291 24891 tablet_service.cc:1467] Processing CreateTablet for tablet d34261daf02f459481b8998e480e4f76 (DEFAULT_TABLE table=test-table [id=0158ef67a9e14641bdf90bf8498e3a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:35.771219 25022 tablet_service.cc:1467] Processing CreateTablet for tablet d34261daf02f459481b8998e480e4f76 (DEFAULT_TABLE table=test-table [id=0158ef67a9e14641bdf90bf8498e3a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:24:35.772174 24760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34261daf02f459481b8998e480e4f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:35.772317 25022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34261daf02f459481b8998e480e4f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:35.772317 24891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34261daf02f459481b8998e480e4f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:35.778363 25105 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Bootstrap starting.
I20260929 16:24:35.779768 25107 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Bootstrap starting.
I20260929 16:24:35.780162 25106 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Bootstrap starting.
I20260929 16:24:35.780791 25105 tablet_bootstrap.cc:654] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:35.781708 25105 log.cc:824] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:35.782295 25106 tablet_bootstrap.cc:654] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:35.783238 25106 log.cc:824] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:35.784121 25105 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: No bootstrap required, opened a new log
I20260929 16:24:35.784339 25105 ts_tablet_manager.cc:1397] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:24:35.784371 25107 tablet_bootstrap.cc:654] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:35.785162 25107 log.cc:824] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:35.785332 25106 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: No bootstrap required, opened a new log
I20260929 16:24:35.785527 25106 ts_tablet_manager.cc:1397] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:24:35.787253 25107 tablet_bootstrap.cc:492] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: No bootstrap required, opened a new log
I20260929 16:24:35.787451 25107 ts_tablet_manager.cc:1397] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:24:35.788046 25105 raft_consensus.cc:348] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.788304 25105 raft_consensus.cc:374] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:35.788379 25105 raft_consensus.cc:729] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf0462b89c94ccaa3cd4a895d1497a3, State: Initialized, Role: FOLLOWER
I20260929 16:24:35.788893 25105 consensus_queue.cc:260] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.789927 25105 ts_tablet_manager.cc:1428] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 16:24:35.789937 25106 raft_consensus.cc:348] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.790196 25106 raft_consensus.cc:374] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:35.790256 25106 raft_consensus.cc:729] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da65efd14430469287c315e8fb7a5790, State: Initialized, Role: FOLLOWER
I20260929 16:24:35.790344 24826 heartbeater.cc:503] Master 127.4.200.126:40525 was elected leader, sending a full tablet report...
I20260929 16:24:35.790724 25106 consensus_queue.cc:260] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.791384 24957 heartbeater.cc:503] Master 127.4.200.126:40525 was elected leader, sending a full tablet report...
I20260929 16:24:35.791711 25106 ts_tablet_manager.cc:1428] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:24:35.791872 25107 raft_consensus.cc:348] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.792124 25107 raft_consensus.cc:374] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:35.792236 25107 raft_consensus.cc:729] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5d954b5474445aac48938be96d1b6e, State: Initialized, Role: FOLLOWER
I20260929 16:24:35.792662 25107 consensus_queue.cc:260] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:35.793388 25088 heartbeater.cc:503] Master 127.4.200.126:40525 was elected leader, sending a full tablet report...
I20260929 16:24:35.793769 25107 ts_tablet_manager.cc:1428] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 16:24:35.846592 24827 tablet.cc:2369] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:24:35.975535 25089 tablet.cc:2369] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:24:36.030778 24958 tablet.cc:2369] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:24:36.206153 25111 raft_consensus.cc:482] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:24:36.206477 25111 raft_consensus.cc:504] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:36.207657 25111 leader_election.cc:288] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5d954b5474445aac48938be96d1b6e (127.4.200.67:44927), da65efd14430469287c315e8fb7a5790 (127.4.200.66:46721)
I20260929 16:24:36.212369 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.208460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:52963 (local address 127.4.200.66:46721)
0929 16:24:36.208608 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:36.208617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:36.208636 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:24:36.208792 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:36.208796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:36.208845 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:36.208927 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:36.209203 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.209766 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211159 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.211726 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211842 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:36.211889 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:36.212161 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:24:36.212236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:24:36.212369 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.208101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.200.65:52043 (local address 127.4.200.67:44927)
0929 16:24:36.208236 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:36.208240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:36.208408 (+ 168us) server_negotiation.cc:407] Connection header received
0929 16:24:36.208477 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:36.208483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:36.208563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:36.208680 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:24:36.208954 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.209480 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211030 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.211656 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211793 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:36.211845 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:36.212161 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 16:24:36.212236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:24:36.212455 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.207945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.67:44927 (local address 127.4.200.65:52043)
0929 16:24:36.208220 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:36.208237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:36.208356 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:36.208696 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:36.208700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:36.208720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:36.208842 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:36.208848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.209599 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.209605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:36.210910 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:36.210920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.211805 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:36.211927 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:36.212047 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 16:24:36.212128 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:24:36.212260 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":69,"thread_start_us":100,"threads_started":1}
I20260929 16:24:36.212455 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.208340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.66:46721 (local address 127.4.200.65:52963)
0929 16:24:36.208581 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:36.208595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:24:36.208676 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:24:36.208938 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:24:36.208941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:24:36.208956 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:24:36.209093 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:36.209098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.209877 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.209880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:36.211049 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:24:36.211057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.211837 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.211840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:24:36.211957 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:36.212047 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 16:24:36.212128 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:24:36.212261 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":96,"thread_start_us":59,"threads_started":1}
I20260929 16:24:36.213256 25042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34261daf02f459481b8998e480e4f76" candidate_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5d954b5474445aac48938be96d1b6e" is_pre_election: true
I20260929 16:24:36.213624 25042 raft_consensus.cc:2454] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf0462b89c94ccaa3cd4a895d1497a3 in term 0.
I20260929 16:24:36.213681 24911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34261daf02f459481b8998e480e4f76" candidate_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da65efd14430469287c315e8fb7a5790" is_pre_election: true
I20260929 16:24:36.214056 24911 raft_consensus.cc:2454] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf0462b89c94ccaa3cd4a895d1497a3 in term 0.
I20260929 16:24:36.214174 24715 leader_election.cc:302] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf0462b89c94ccaa3cd4a895d1497a3, bb5d954b5474445aac48938be96d1b6e; no voters:
I20260929 16:24:36.214514 25111 raft_consensus.cc:2781] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:24:36.214612 25111 raft_consensus.cc:482] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:24:36.214675 25111 raft_consensus.cc:3037] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:36.215698 25111 raft_consensus.cc:504] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:36.216049 25111 leader_election.cc:288] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [CANDIDATE]: Term 1 election: Requested vote from peers bb5d954b5474445aac48938be96d1b6e (127.4.200.67:44927), da65efd14430469287c315e8fb7a5790 (127.4.200.66:46721)
I20260929 16:24:36.216432 25042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34261daf02f459481b8998e480e4f76" candidate_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5d954b5474445aac48938be96d1b6e"
I20260929 16:24:36.216457 24911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34261daf02f459481b8998e480e4f76" candidate_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da65efd14430469287c315e8fb7a5790"
I20260929 16:24:36.216559 24911 raft_consensus.cc:3037] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:36.216559 25042 raft_consensus.cc:3037] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:36.217803 25042 raft_consensus.cc:2454] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf0462b89c94ccaa3cd4a895d1497a3 in term 1.
I20260929 16:24:36.217803 24911 raft_consensus.cc:2454] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf0462b89c94ccaa3cd4a895d1497a3 in term 1.
I20260929 16:24:36.218101 24715 leader_election.cc:302] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf0462b89c94ccaa3cd4a895d1497a3, bb5d954b5474445aac48938be96d1b6e; no voters:
I20260929 16:24:36.218312 25111 raft_consensus.cc:2781] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:36.218573 25111 raft_consensus.cc:686] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf0462b89c94ccaa3cd4a895d1497a3, State: Running, Role: LEADER
I20260929 16:24:36.218922 25111 consensus_queue.cc:237] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } }
I20260929 16:24:36.222476 24637 catalog_manager.cc:5495] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bf0462b89c94ccaa3cd4a895d1497a3 (127.4.200.65). New cstate: current_term: 1 leader_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf0462b89c94ccaa3cd4a895d1497a3" member_type: VOTER last_known_addr { host: "127.4.200.65" port: 35951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 } health_report { overall_health: UNKNOWN } } }
I20260929 16:24:36.328480 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.325092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.4.200.65:35951)
0929 16:24:36.325331 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:36.325334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:36.325349 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:24:36.325455 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:36.325457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:36.325511 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:36.325611 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:24:36.325831 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.326344 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.327189 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:36.327405 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:36.327481 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:36.327765 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:24:36.328159 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:24:36.328245 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:36.328315 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:24:36.328359 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260929 16:24:36.333997 24911 raft_consensus.cc:1261] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 1 FOLLOWER]: Refusing update from remote peer 4bf0462b89c94ccaa3cd4a895d1497a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:36.333997 25042 raft_consensus.cc:1261] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 1 FOLLOWER]: Refusing update from remote peer 4bf0462b89c94ccaa3cd4a895d1497a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:24:36.334725 25111 consensus_queue.cc:1035] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5d954b5474445aac48938be96d1b6e" member_type: VOTER last_known_addr { host: "127.4.200.67" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:36.334908 25116 consensus_queue.cc:1035] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da65efd14430469287c315e8fb7a5790" member_type: VOTER last_known_addr { host: "127.4.200.66" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:24:36.342999 25124 mvcc.cc:205] Tried to move back new op lower bound from 7334703416656797696 to 7334703416194998272. Current Snapshot: MvccSnapshot[applied={T|T < 7334703416656797696}]
I20260929 16:24:36.351511 24638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:36.351693 24638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:24:36.354244 24638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 190eaa8174ee4516a71e7eea0b65c8e2: Sending DeleteTablet for 3 replicas of tablet d34261daf02f459481b8998e480e4f76
I20260929 16:24:36.354967 24760 tablet_service.cc:1514] Processing DeleteTablet for tablet d34261daf02f459481b8998e480e4f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:36 UTC) from {username='slave'} at 127.0.0.1:48590
I20260929 16:24:36.355136 25022 tablet_service.cc:1514] Processing DeleteTablet for tablet d34261daf02f459481b8998e480e4f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:36 UTC) from {username='slave'} at 127.0.0.1:48750
I20260929 16:24:36.355218 24891 tablet_service.cc:1514] Processing DeleteTablet for tablet d34261daf02f459481b8998e480e4f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:24:36 UTC) from {username='slave'} at 127.0.0.1:44368
I20260929 16:24:36.356066 25133 tablet_replica.cc:330] T d34261daf02f459481b8998e480e4f76 P 4bf0462b89c94ccaa3cd4a895d1497a3: stopping tablet replica
I20260929 16:24:36.356341 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24698
I20260929 16:24:36.357270 25135 tablet_replica.cc:330] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: stopping tablet replica
I20260929 16:24:36.357564 25135 raft_consensus.cc:2229] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:36.357868 25135 raft_consensus.cc:2258] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:36.358263 25134 tablet_replica.cc:330] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: stopping tablet replica
I20260929 16:24:36.358594 25134 raft_consensus.cc:2229] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:24:36.358870 25134 raft_consensus.cc:2258] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:24:36.359251 25135 ts_tablet_manager.cc:1905] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:36.360064 25134 ts_tablet_manager.cc:1905] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:24:36.362098 25135 ts_tablet_manager.cc:1918] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:36.362193 25135 log.cc:1196] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-1/wal/wals/d34261daf02f459481b8998e480e4f76
I20260929 16:24:36.362564 25135 ts_tablet_manager.cc:1939] T d34261daf02f459481b8998e480e4f76 P da65efd14430469287c315e8fb7a5790: Deleting consensus metadata
I20260929 16:24:36.362771 25134 ts_tablet_manager.cc:1918] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:24:36.362860 25134 log.cc:1196] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Deleting WAL directory at /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790698821455595-4897-0/minicluster-data/ts-2/wal/wals/d34261daf02f459481b8998e480e4f76
W20260929 16:24:36.363147 24622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.200.65:35951 (error 108)
I20260929 16:24:36.363595 25134 ts_tablet_manager.cc:1939] T d34261daf02f459481b8998e480e4f76 P bb5d954b5474445aac48938be96d1b6e: Deleting consensus metadata
I20260929 16:24:36.363822 24624 catalog_manager.cc:4841] TS da65efd14430469287c315e8fb7a5790 (127.4.200.66:46721): tablet d34261daf02f459481b8998e480e4f76 (table test-table [id=0158ef67a9e14641bdf90bf8498e3a11]) successfully deleted
I20260929 16:24:36.364185 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:36.363775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.65:35951 (local address 127.0.0.1:48596)
0929 16:24:36.363986 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:24:36.364073 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.65:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260929 16:24:36.364231 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24829
W20260929 16:24:36.365191 24622 catalog_manager.cc:4584] TS 4bf0462b89c94ccaa3cd4a895d1497a3 (127.4.200.65:35951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d34261daf02f459481b8998e480e4f76: Network error: Client connection negotiation failed: client connection to 127.4.200.65:35951: connect: Connection refused (error 111)
I20260929 16:24:36.365681 24624 catalog_manager.cc:4841] TS bb5d954b5474445aac48938be96d1b6e (127.4.200.67:44927): tablet d34261daf02f459481b8998e480e4f76 (table test-table [id=0158ef67a9e14641bdf90bf8498e3a11]) successfully deleted
I20260929 16:24:36.370200 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24960
I20260929 16:24:36.376286 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 24607
2026-09-29T16:24:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1482 ms)
[----------] 6 tests from EncryptionPolicyTest (10910 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:36 dist-test-slave-s9ct krb5kdc[25140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:36 dist-test-slave-s9ct krb5kdc[25140](info): set up 2 sockets
Sep 29 16:24:36 dist-test-slave-s9ct krb5kdc[25140](info): commencing operation
krb5kdc: starting...
W20260929 16:24:37.410671 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:37.541235 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:37.649046 25153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:37.649302 25153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:37.649343 25153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:37.653837 25153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:37.653888 25153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:37.653908 25153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:37.653923 25153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:37.653937 25153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:37.659013 25153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:37.659873 25153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:37.660758 25153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:37.668807 25159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:37.668795 25161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:37.668977 25153 server_base.cc:1034] running on GCE node
W20260929 16:24:37.668875 25158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:37.669567 25153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:37.670692 25153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:37.671871 25153 hybrid_clock.cc:648] HybridClock initialized: now 1790699077671854 us; error 29 us; skew 500 ppm
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:37.674544 25153 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:37.675384 25153 webserver.cc:460] Webserver started at http://127.0.0.1:43333/ using document root <none> and password file <none>
I20260929 16:24:37.675916 25153 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:37.676007 25153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:37.676219 25153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:37.678207 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "44df643f803d45cc8cce69bf181e155c"
format_stamp: "Formatted at 2026-09-29 16:24:37 on dist-test-slave-s9ct"
server_key: "69b54052315457a5701df696a5b9836f"
server_key_iv: "bcf6446848e814c803a6620967122d84"
server_key_version: "encryptionkey@0"
I20260929 16:24:37.678817 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "44df643f803d45cc8cce69bf181e155c"
format_stamp: "Formatted at 2026-09-29 16:24:37 on dist-test-slave-s9ct"
server_key: "69b54052315457a5701df696a5b9836f"
server_key_iv: "bcf6446848e814c803a6620967122d84"
server_key_version: "encryptionkey@0"
I20260929 16:24:37.682257 25153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:37.684660 25168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:37.685814 25153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:37.685956 25153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "44df643f803d45cc8cce69bf181e155c"
format_stamp: "Formatted at 2026-09-29 16:24:37 on dist-test-slave-s9ct"
server_key: "69b54052315457a5701df696a5b9836f"
server_key_iv: "bcf6446848e814c803a6620967122d84"
server_key_version: "encryptionkey@0"
I20260929 16:24:37.686064 25153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:37.719858 25153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:37.723237 25153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:37.723438 25153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:37.730798 25153 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46827
I20260929 16:24:37.730823 25220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46827 every 8 connection(s)
I20260929 16:24:37.731711 25153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:37.734328 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:37.737429 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25153
I20260929 16:24:37.737547 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:37.737843 4897 external_mini_cluster.cc:1428] Setting key 439f6a781b7e7d8f5a37dcbc8f93a945
I20260929 16:24:37.740453 25221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: Bootstrap starting.
I20260929 16:24:37.742738 25221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:37.743484 25221 log.cc:824] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:37.745337 25221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: No bootstrap required, opened a new log
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:24:37.748837 25221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260929 16:24:37.749063 25221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:37.749142 25221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44df643f803d45cc8cce69bf181e155c, State: Initialized, Role: FOLLOWER
I20260929 16:24:37.749588 25221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260929 16:24:37.749708 25221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:37.749795 25221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:37.749904 25221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:37.751114 25221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260929 16:24:37.751464 25221 leader_election.cc:302] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44df643f803d45cc8cce69bf181e155c; no voters:
I20260929 16:24:37.751773 25221 leader_election.cc:288] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:37.751885 25226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:37.752105 25226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [term 1 LEADER]: Becoming Leader. State: Replica: 44df643f803d45cc8cce69bf181e155c, State: Running, Role: LEADER
I20260929 16:24:37.752498 25226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260929 16:24:37.753108 25221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:37.753738 25228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44df643f803d45cc8cce69bf181e155c. Latest consensus state: current_term: 1 leader_uuid: "44df643f803d45cc8cce69bf181e155c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260929 16:24:37.753784 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:37.739309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60407 (local address 127.0.0.1:46827)
0929 16:24:37.739760 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:37.739775 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:37.739809 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:24:37.740421 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:37.740438 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:37.740828 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:37.741169 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:37.741949 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:37.743247 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:37.744144 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:37.744607 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:37.744780 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:37.747613 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:37.747619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:37.747633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:37.747662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:37.750180 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:37.750755 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:37.750760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:37.750764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:37.750875 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:37.751289 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:37.751292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:37.751294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:37.751765 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:37.751997 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:37.752495 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 16:24:37.752729 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260929 16:24:37.753867 25228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:37.755514 25227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44df643f803d45cc8cce69bf181e155c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df643f803d45cc8cce69bf181e155c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260929 16:24:37.756309 25227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:37.757282 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 16:24:37.758935 25241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:37.759013 25241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:37.759073 25233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:37.760007 25233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:37.765372 25233 catalog_manager.cc:1347] Generated new cluster ID: d555904b5d9741d58e4bdd771cea8b07
I20260929 16:24:37.765431 25233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:37.768682 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:37.759411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50555 (local address 127.0.0.1:46827)
0929 16:24:37.759513 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:37.759518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:37.759749 (+ 231us) server_negotiation.cc:407] Connection header received
0929 16:24:37.759851 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:37.759854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:37.759903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:37.759991 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:37.760785 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:37.761596 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:37.762857 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:37.763189 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:37.763297 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:37.765250 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:37.765255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:37.765259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:37.765297 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:37.766903 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:37.767391 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:37.767394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:37.767396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:37.767475 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:37.767803 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:37.767806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:37.767808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:37.767972 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:37.768126 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:37.768397 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:24:37.768490 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:24:37.779603 25233 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:37.780747 25233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:37.785073 25233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44df643f803d45cc8cce69bf181e155c: Generated new TSK 0
I20260929 16:24:37.785817 25233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:37.790841 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25153
W20260929 16:24:37.796501 25138 connection.cc:535] client connection to 127.0.0.1:46827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46827): Connection reset by peer (error 104)
2026-09-29T16:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25253](info): set up 2 sockets
Sep 29 16:24:37 dist-test-slave-s9ct krb5kdc[25253](info): commencing operation
krb5kdc: starting...
W20260929 16:24:38.826746 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:38 dist-test-slave-s9ct krb5kdc[25253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:38.950582 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:39.058557 25266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:39.058800 25266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:39.058840 25266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:39.062810 25266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:39.062860 25266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:39.062879 25266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:39.062896 25266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:39.062911 25266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:39.068120 25266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:39.069100 25266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:39.070123 25266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:39.078313 25271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:39.078346 25274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:39.078379 25272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:39.078516 25266 server_base.cc:1034] running on GCE node
I20260929 16:24:39.079218 25266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:39.080227 25266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:39.081414 25266 hybrid_clock.cc:648] HybridClock initialized: now 1790699079081368 us; error 54 us; skew 500 ppm
Sep 29 16:24:39 dist-test-slave-s9ct krb5kdc[25253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:39.084345 25266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:39.085202 25266 webserver.cc:460] Webserver started at http://127.0.0.1:43119/ using document root <none> and password file <none>
I20260929 16:24:39.085812 25266 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:39.085876 25266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:39.086100 25266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:39.088338 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "fe1f230846d94f6380899b0fcd976e2f"
format_stamp: "Formatted at 2026-09-29 16:24:39 on dist-test-slave-s9ct"
server_key: "0d774092a83a367a76378b5fc9f877cd"
server_key_iv: "bb5a84bcbdfa530944e8350961f6ae63"
server_key_version: "encryptionkey@0"
I20260929 16:24:39.088950 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "fe1f230846d94f6380899b0fcd976e2f"
format_stamp: "Formatted at 2026-09-29 16:24:39 on dist-test-slave-s9ct"
server_key: "0d774092a83a367a76378b5fc9f877cd"
server_key_iv: "bb5a84bcbdfa530944e8350961f6ae63"
server_key_version: "encryptionkey@0"
I20260929 16:24:39.092944 25266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:24:39.095681 25281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:39.096958 25266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 16:24:39.097118 25266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "fe1f230846d94f6380899b0fcd976e2f"
format_stamp: "Formatted at 2026-09-29 16:24:39 on dist-test-slave-s9ct"
server_key: "0d774092a83a367a76378b5fc9f877cd"
server_key_iv: "bb5a84bcbdfa530944e8350961f6ae63"
server_key_version: "encryptionkey@0"
I20260929 16:24:39.097246 25266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:39.109408 25266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:39.112506 25266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:39.112748 25266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:39.120275 25266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38527
I20260929 16:24:39.120317 25333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38527 every 8 connection(s)
I20260929 16:24:39.121275 25266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:39.124190 25334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:39.126183 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25266
I20260929 16:24:39.126348 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:39.126693 4897 external_mini_cluster.cc:1428] Setting key 275d6ab882101c505c1da175e3d25de7
I20260929 16:24:39.130964 25334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: Bootstrap starting.
I20260929 16:24:39.133235 25334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:39.134181 25334 log.cc:824] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:39.135725 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:39.128120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33095 (local address 127.0.0.1:38527)
0929 16:24:39.128580 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:39.128596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:39.128634 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:24:39.129295 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:39.129311 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:39.129633 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:39.129971 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:39.130371 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:39.131475 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:39.132400 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:39.132864 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:39.133063 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:39.133980 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:39.133985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:39.133996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:39.134483 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:39.134540 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:39.134565 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:39.134724 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:39.134913 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:24:39.135042 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260929 16:24:39.136319 25334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: No bootstrap required, opened a new log
I20260929 16:24:39.138777 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:39.140825 25334 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260929 16:24:39.141137 25334 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:39.141211 25334 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1f230846d94f6380899b0fcd976e2f, State: Initialized, Role: FOLLOWER
I20260929 16:24:39.141731 25334 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260929 16:24:39.141867 25334 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:39.141963 25334 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:39.142071 25334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:39.143291 25334 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260929 16:24:39.143682 25334 leader_election.cc:302] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1f230846d94f6380899b0fcd976e2f; no voters:
I20260929 16:24:39.144016 25334 leader_election.cc:288] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:39.144135 25344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:39.144376 25344 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [term 1 LEADER]: Becoming Leader. State: Replica: fe1f230846d94f6380899b0fcd976e2f, State: Running, Role: LEADER
I20260929 16:24:39.144731 25344 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260929 16:24:39.145501 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:39.140768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32813 (local address 127.0.0.1:38527)
0929 16:24:39.140880 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:39.140884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:39.141028 (+ 144us) server_negotiation.cc:407] Connection header received
0929 16:24:39.141102 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:39.141105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:39.141155 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:39.141249 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:39.141644 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:39.142303 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:39.143574 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:39.143847 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:39.143925 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:39.144864 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:39.144868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:39.144871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:39.144883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:39.144910 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:39.144948 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:39.145049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:39.145353 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:24:39.145386 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 16:24:39.145533 25334 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:39.148286 25345 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1f230846d94f6380899b0fcd976e2f. Latest consensus state: current_term: 1 leader_uuid: "fe1f230846d94f6380899b0fcd976e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } } }
I20260929 16:24:39.148425 25345 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:39.148269 25346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1f230846d94f6380899b0fcd976e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f230846d94f6380899b0fcd976e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } } }
I20260929 16:24:39.149078 25346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f [sys.catalog]: This master's current role is: LEADER
W20260929 16:24:39.151439 25360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:39.151506 25360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:39.151630 25359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:39.152479 25359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:39.158726 25359 catalog_manager.cc:1347] Generated new cluster ID: 1988349c9aec45248b340168892ff8a2
I20260929 16:24:39.158797 25359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:39.178290 25359 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:39.179455 25359 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:39.185111 25359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1f230846d94f6380899b0fcd976e2f: Generated new TSK 0
I20260929 16:24:39.185839 25359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:39.189719 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25266
W20260929 16:24:39.195137 25251 connection.cc:535] client connection to 127.0.0.1:38527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38527): Connection reset by peer (error 104)
2026-09-29T16:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:39 dist-test-slave-s9ct krb5kdc[25365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:39 dist-test-slave-s9ct krb5kdc[25365](info): set up 2 sockets
Sep 29 16:24:39 dist-test-slave-s9ct krb5kdc[25365](info): commencing operation
krb5kdc: starting...
W20260929 16:24:40.243955 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:40 dist-test-slave-s9ct krb5kdc[25365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:40.374020 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:40.485170 25378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:40.485438 25378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:40.485482 25378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:40.489647 25378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:40.489711 25378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:40.489732 25378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:40.489748 25378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:40.489790 25378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:40.495079 25378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:40.496176 25378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:40.497201 25378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:40.505909 25384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:40.506004 25378 server_base.cc:1034] running on GCE node
W20260929 16:24:40.506170 25383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:40.506176 25386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:40.507009 25378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:40.508380 25378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:40.509615 25378 hybrid_clock.cc:648] HybridClock initialized: now 1790699080509532 us; error 87 us; skew 500 ppm
Sep 29 16:24:40 dist-test-slave-s9ct krb5kdc[25365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:40.512607 25378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:40.513548 25378 webserver.cc:460] Webserver started at http://127.0.0.1:33787/ using document root <none> and password file <none>
I20260929 16:24:40.514110 25378 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:40.514171 25378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:40.514431 25378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:40.516515 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "cf67ce63f8524a40b20f2bfa9b221abc"
format_stamp: "Formatted at 2026-09-29 16:24:40 on dist-test-slave-s9ct"
server_key: "d96dce12b9121d4f6876ce103ee8f40e"
server_key_iv: "e7d2aa739eeaf1c3d852b35f0e7d9b4b"
server_key_version: "encryptionkey@0"
I20260929 16:24:40.517203 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "cf67ce63f8524a40b20f2bfa9b221abc"
format_stamp: "Formatted at 2026-09-29 16:24:40 on dist-test-slave-s9ct"
server_key: "d96dce12b9121d4f6876ce103ee8f40e"
server_key_iv: "e7d2aa739eeaf1c3d852b35f0e7d9b4b"
server_key_version: "encryptionkey@0"
I20260929 16:24:40.521338 25378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:40.523912 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:40.525157 25378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:40.525316 25378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "cf67ce63f8524a40b20f2bfa9b221abc"
format_stamp: "Formatted at 2026-09-29 16:24:40 on dist-test-slave-s9ct"
server_key: "d96dce12b9121d4f6876ce103ee8f40e"
server_key_iv: "e7d2aa739eeaf1c3d852b35f0e7d9b4b"
server_key_version: "encryptionkey@0"
I20260929 16:24:40.525429 25378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:40.530108 25378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:40.533388 25378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:40.533591 25378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:40.540977 25378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37703
I20260929 16:24:40.540995 25445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37703 every 8 connection(s)
I20260929 16:24:40.541931 25378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:40.544723 25446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:40.550174 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25378
I20260929 16:24:40.550307 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:40.550359 25446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: Bootstrap starting.
I20260929 16:24:40.550717 4897 external_mini_cluster.cc:1428] Setting key f347e43893383765425ce43a14c2de24
I20260929 16:24:40.552942 25446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:40.553681 25446 log.cc:824] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:40.555753 25446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: No bootstrap required, opened a new log
I20260929 16:24:40.559860 25446 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260929 16:24:40.560097 25446 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:40.560190 25446 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf67ce63f8524a40b20f2bfa9b221abc, State: Initialized, Role: FOLLOWER
I20260929 16:24:40.560672 25446 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260929 16:24:40.560873 25446 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:40.560952 25446 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:40.561151 25446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:40.561913 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:40.552114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39569 (local address 127.0.0.1:37703)
0929 16:24:40.552619 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:40.552635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:40.552672 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:24:40.553386 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:40.553405 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:40.553709 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:40.554103 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:40.560525 (+ 6422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:40.560531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:40.560547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:40.560566 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:40.560628 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:40.560650 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:40.560753 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:40.560945 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 16:24:40.561115 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260929 16:24:40.562201 25446 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260929 16:24:40.562600 25446 leader_election.cc:302] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf67ce63f8524a40b20f2bfa9b221abc; no voters:
I20260929 16:24:40.562963 25446 leader_election.cc:288] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:40.563048 25451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:40.563330 25451 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [term 1 LEADER]: Becoming Leader. State: Replica: cf67ce63f8524a40b20f2bfa9b221abc, State: Running, Role: LEADER
I20260929 16:24:40.563678 25451 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260929 16:24:40.564368 25446 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:40.565050 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:40.565656 25452 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } } }
I20260929 16:24:40.565789 25452 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:40.566082 25453 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf67ce63f8524a40b20f2bfa9b221abc. Latest consensus state: current_term: 1 leader_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf67ce63f8524a40b20f2bfa9b221abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } } }
I20260929 16:24:40.566164 25453 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:40.569208 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:40.567450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54751 (local address 127.0.0.1:37703)
0929 16:24:40.567565 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:40.567572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:40.567828 (+ 256us) server_negotiation.cc:407] Connection header received
0929 16:24:40.567934 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:40.567936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:40.567971 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:40.568058 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:40.568816 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:40.568819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:40.568822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:40.568834 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:40.568862 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:40.568875 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:40.568961 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:40.569059 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:24:40.569095 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 16:24:40.570254 25471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:40.570345 25471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:40.570448 25460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:40.571130 25460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:40.576972 25460 catalog_manager.cc:1347] Generated new cluster ID: 6aa30275c4334e75a3eb4c6b1ef28de9
I20260929 16:24:40.577059 25460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:40.604933 25460 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:40.605296 25460 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:40.611416 25460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf67ce63f8524a40b20f2bfa9b221abc: Generated new TSK 0
I20260929 16:24:40.612080 25460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:40.621673 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25378
2026-09-29T16:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:40 dist-test-slave-s9ct krb5kdc[25477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:40 dist-test-slave-s9ct krb5kdc[25477](info): set up 2 sockets
Sep 29 16:24:40 dist-test-slave-s9ct krb5kdc[25477](info): commencing operation
krb5kdc: starting...
W20260929 16:24:41.682834 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:41 dist-test-slave-s9ct krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:41.810384 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:41.920257 25490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:41.920557 25490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:41.920601 25490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:41.925364 25490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:41.925439 25490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:41.925460 25490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:41.925477 25490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:41.925493 25490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:41.931042 25490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:41.932054 25490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:41.933032 25490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:41.941792 25498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:41.941723 25496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:41.941728 25495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:41.941931 25490 server_base.cc:1034] running on GCE node
I20260929 16:24:41.942766 25490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:41.943842 25490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:41.945019 25490 hybrid_clock.cc:648] HybridClock initialized: now 1790699081944999 us; error 30 us; skew 500 ppm
Sep 29 16:24:41 dist-test-slave-s9ct krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:41.947968 25490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:41.948869 25490 webserver.cc:460] Webserver started at http://127.0.0.1:37085/ using document root <none> and password file <none>
I20260929 16:24:41.949436 25490 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:41.949497 25490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:41.949738 25490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:41.951864 25490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "281143a41b634a34b67f48dbef89a691"
format_stamp: "Formatted at 2026-09-29 16:24:41 on dist-test-slave-s9ct"
server_key: "f114b440f690146a8d02a5b865f9a3e0"
server_key_iv: "02e7def988c4312d173477e0e90d20b3"
server_key_version: "encryptionkey@0"
I20260929 16:24:41.952414 25490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "281143a41b634a34b67f48dbef89a691"
format_stamp: "Formatted at 2026-09-29 16:24:41 on dist-test-slave-s9ct"
server_key: "f114b440f690146a8d02a5b865f9a3e0"
server_key_iv: "02e7def988c4312d173477e0e90d20b3"
server_key_version: "encryptionkey@0"
I20260929 16:24:41.956256 25490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:24:41.958791 25505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:41.960036 25490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:41.960210 25490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "281143a41b634a34b67f48dbef89a691"
format_stamp: "Formatted at 2026-09-29 16:24:41 on dist-test-slave-s9ct"
server_key: "f114b440f690146a8d02a5b865f9a3e0"
server_key_iv: "02e7def988c4312d173477e0e90d20b3"
server_key_version: "encryptionkey@0"
I20260929 16:24:41.960333 25490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:41.972127 25490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:41.975075 25490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:41.975284 25490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:41.982579 25490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43157
I20260929 16:24:41.982616 25557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43157 every 8 connection(s)
I20260929 16:24:41.983560 25490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:41.986322 25558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:41.986683 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25490
I20260929 16:24:41.986824 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:41.987129 4897 external_mini_cluster.cc:1428] Setting key db3e9e6adcba3e40a7288f924fd389ca
I20260929 16:24:41.993467 25558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: Bootstrap starting.
I20260929 16:24:41.996053 25558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: Neither blocks nor log segments found. Creating new log.
Sep 29 16:24:41 dist-test-slave-s9ct krb5kdc[25477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:24:41.996850 25558 log.cc:824] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:41.998835 25558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: No bootstrap required, opened a new log
I20260929 16:24:42.001820 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:41.988510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47501 (local address 127.0.0.1:43157)
0929 16:24:41.988935 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:41.988954 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:41.988997 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:24:41.989740 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:41.989765 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:41.990202 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:41.990578 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:41.991252 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:41.992771 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:41.993747 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:41.994424 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:41.994586 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:41.997164 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:41.997172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:41.997194 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:41.997229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:41.999063 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:41.999530 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:41.999535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:41.999539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:41.999642 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:41.999946 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:41.999948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:41.999950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:42.000323 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:42.000457 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:42.000799 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:24:42.001095 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260929 16:24:42.003006 25558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260929 16:24:42.003260 25558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:42.003340 25558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 281143a41b634a34b67f48dbef89a691, State: Initialized, Role: FOLLOWER
I20260929 16:24:42.003808 25558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260929 16:24:42.003970 25558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:42.004038 25558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:42.004149 25558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:42.004711 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:42.005214 25558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260929 16:24:42.005573 25558 leader_election.cc:302] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 281143a41b634a34b67f48dbef89a691; no voters:
I20260929 16:24:42.005913 25558 leader_election.cc:288] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:42.005982 25564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:42.006240 25564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [term 1 LEADER]: Becoming Leader. State: Replica: 281143a41b634a34b67f48dbef89a691, State: Running, Role: LEADER
I20260929 16:24:42.006598 25564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260929 16:24:42.007318 25558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:42.008462 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "281143a41b634a34b67f48dbef89a691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260929 16:24:42.008725 25569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 281143a41b634a34b67f48dbef89a691. Latest consensus state: current_term: 1 leader_uuid: "281143a41b634a34b67f48dbef89a691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281143a41b634a34b67f48dbef89a691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260929 16:24:42.008881 25569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:42.009143 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691 [sys.catalog]: This master's current role is: LEADER
W20260929 16:24:42.012656 25583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:42.012740 25583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:42.012791 25579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:42.013595 25579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:42.016179 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:42.006787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50933 (local address 127.0.0.1:43157)
0929 16:24:42.006916 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:42.006921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:42.007213 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:24:42.007366 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:42.007369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:42.007421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:42.007469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:42.008328 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:42.009119 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:42.010310 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:42.010595 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:42.010687 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:42.012322 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:42.012327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:42.012330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:42.012355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:42.014128 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:42.014645 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:42.014649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:42.014651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:42.014737 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:42.015049 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:42.015053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:42.015055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:42.015265 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:42.015409 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:42.015757 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:24:42.015872 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:24:42.019353 25579 catalog_manager.cc:1347] Generated new cluster ID: 04fca8631dff464c95b4e00e6f82f4a4
I20260929 16:24:42.019418 25579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:42.047595 25579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:42.048575 25579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:42.057679 25579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 281143a41b634a34b67f48dbef89a691: Generated new TSK 0
I20260929 16:24:42.058455 25579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:42.065964 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25490
2026-09-29T16:24:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:42 dist-test-slave-s9ct krb5kdc[25589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:42 dist-test-slave-s9ct krb5kdc[25589](info): set up 2 sockets
Sep 29 16:24:42 dist-test-slave-s9ct krb5kdc[25589](info): commencing operation
krb5kdc: starting...
W20260929 16:24:43.114807 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:43 dist-test-slave-s9ct krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:43.238358 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:43.347324 25602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:43.347592 25602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:43.347636 25602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:43.351581 25602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:43.351637 25602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:43.351656 25602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:43.351675 25602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:43.351691 25602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:43.356884 25602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:43.357913 25602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:43.358906 25602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:43.367472 25610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:43.367446 25608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:43.367446 25607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:43.367904 25602 server_base.cc:1034] running on GCE node
I20260929 16:24:43.368479 25602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:43.369624 25602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:43.370824 25602 hybrid_clock.cc:648] HybridClock initialized: now 1790699083370806 us; error 31 us; skew 500 ppm
Sep 29 16:24:43 dist-test-slave-s9ct krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:43.373503 25602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:43.374524 25602 webserver.cc:460] Webserver started at http://127.0.0.1:35497/ using document root <none> and password file <none>
I20260929 16:24:43.375111 25602 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:43.375202 25602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:43.375434 25602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:43.377426 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "d169f89692d546f19901f44d67a43e6d"
format_stamp: "Formatted at 2026-09-29 16:24:43 on dist-test-slave-s9ct"
server_key: "e584c7b01a661c89edc22762d9a7b5bf"
server_key_iv: "b6ee0c7b40fe61a9d65d0a3b10aeac7a"
server_key_version: "encryptionkey@0"
I20260929 16:24:43.377985 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "d169f89692d546f19901f44d67a43e6d"
format_stamp: "Formatted at 2026-09-29 16:24:43 on dist-test-slave-s9ct"
server_key: "e584c7b01a661c89edc22762d9a7b5bf"
server_key_iv: "b6ee0c7b40fe61a9d65d0a3b10aeac7a"
server_key_version: "encryptionkey@0"
I20260929 16:24:43.381631 25602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:24:43.384014 25617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:43.385382 25602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:43.385478 25602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "d169f89692d546f19901f44d67a43e6d"
format_stamp: "Formatted at 2026-09-29 16:24:43 on dist-test-slave-s9ct"
server_key: "e584c7b01a661c89edc22762d9a7b5bf"
server_key_iv: "b6ee0c7b40fe61a9d65d0a3b10aeac7a"
server_key_version: "encryptionkey@0"
I20260929 16:24:43.385564 25602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:43.412163 25602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:43.414999 25602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:43.415161 25602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:43.422082 25602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39327
I20260929 16:24:43.422108 25669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39327 every 8 connection(s)
I20260929 16:24:43.423044 25602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:43.424618 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25602
I20260929 16:24:43.424773 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:43.425055 4897 external_mini_cluster.cc:1428] Setting key cfaeed9a304c36a3c7e80d48f38d9f95
I20260929 16:24:43.426103 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:43.431537 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d: Bootstrap starting.
I20260929 16:24:43.433800 25670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:43.434033 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:43.426517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50849 (local address 127.0.0.1:39327)
0929 16:24:43.426952 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:43.426967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:43.427000 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:24:43.427638 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:43.427653 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:43.427976 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:43.428310 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:43.428704 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:43.429690 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:43.430814 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:43.431284 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:43.431464 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:43.432377 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:43.432382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:43.432394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:43.432763 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:43.432818 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:43.432840 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:43.432953 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:43.433110 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 16:24:43.433302 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20260929 16:24:43.434645 25670 log.cc:824] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:43.436614 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d: No bootstrap required, opened a new log
I20260929 16:24:43.436853 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:43.440820 25670 raft_consensus.cc:348] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } }
I20260929 16:24:43.441079 25670 raft_consensus.cc:374] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:43.441141 25670 raft_consensus.cc:729] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d169f89692d546f19901f44d67a43e6d, State: Initialized, Role: FOLLOWER
I20260929 16:24:43.441568 25670 consensus_queue.cc:260] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } }
I20260929 16:24:43.441730 25670 raft_consensus.cc:388] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:43.441838 25670 raft_consensus.cc:482] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:43.441956 25670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:43.442984 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:43.438720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44057 (local address 127.0.0.1:39327)
0929 16:24:43.438829 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:43.438832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:43.439026 (+ 194us) server_negotiation.cc:407] Connection header received
0929 16:24:43.439144 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:43.439146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:43.439180 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:43.439242 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:43.439604 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:43.440094 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:43.441157 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:43.441579 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:43.441678 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:43.442514 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:43.442518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:43.442519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:43.442530 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:43.442555 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:43.442568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:43.442627 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:43.442770 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:24:43.442814 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:24:43.443217 25670 raft_consensus.cc:504] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } }
I20260929 16:24:43.443567 25670 leader_election.cc:302] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d169f89692d546f19901f44d67a43e6d; no voters:
I20260929 16:24:43.443888 25670 leader_election.cc:288] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:43.444001 25680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:43.444253 25680 raft_consensus.cc:686] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [term 1 LEADER]: Becoming Leader. State: Replica: d169f89692d546f19901f44d67a43e6d, State: Running, Role: LEADER
I20260929 16:24:43.444587 25680 consensus_queue.cc:237] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } }
I20260929 16:24:43.445055 25670 sys_catalog.cc:564] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:43.446234 25681 sys_catalog.cc:455] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d169f89692d546f19901f44d67a43e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } } }
I20260929 16:24:43.446376 25681 sys_catalog.cc:458] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:43.446696 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d169f89692d546f19901f44d67a43e6d. Latest consensus state: current_term: 1 leader_uuid: "d169f89692d546f19901f44d67a43e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d169f89692d546f19901f44d67a43e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39327 } } }
I20260929 16:24:43.446789 25689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:43.446815 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:43.449478 25689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:43.455049 25689 catalog_manager.cc:1347] Generated new cluster ID: f02d709c3e83403eb0cfdedc56927bc3
I20260929 16:24:43.455120 25689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:43.466777 25689 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:43.467993 25689 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:43.485939 25689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d169f89692d546f19901f44d67a43e6d: Generated new TSK 0
I20260929 16:24:43.486938 25689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:43.498292 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25602
2026-09-29T16:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:43 dist-test-slave-s9ct krb5kdc[25701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:43 dist-test-slave-s9ct krb5kdc[25701](info): set up 2 sockets
Sep 29 16:24:43 dist-test-slave-s9ct krb5kdc[25701](info): commencing operation
krb5kdc: starting...
W20260929 16:24:44.534783 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:44 dist-test-slave-s9ct krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:44.658255 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:44.767657 25714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:44.767920 25714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:44.767961 25714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:44.771953 25714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:44.772015 25714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:44.772034 25714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:44.772051 25714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:44.772066 25714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:44.778156 25714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:44.779242 25714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:44.780186 25714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:44.789507 25720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:44.789530 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:44.789635 25714 server_base.cc:1034] running on GCE node
W20260929 16:24:44.789513 25719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:44.790386 25714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:44.791472 25714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:44.792678 25714 hybrid_clock.cc:648] HybridClock initialized: now 1790699084792662 us; error 37 us; skew 500 ppm
Sep 29 16:24:44 dist-test-slave-s9ct krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:44.795493 25714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:44.796335 25714 webserver.cc:460] Webserver started at http://127.0.0.1:37553/ using document root <none> and password file <none>
I20260929 16:24:44.796880 25714 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:44.796940 25714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:44.797184 25714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:44.799279 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b"
format_stamp: "Formatted at 2026-09-29 16:24:44 on dist-test-slave-s9ct"
server_key: "f1981b293d9f507bd49c5be486bdd74a"
server_key_iv: "f921135bc8055fefb8f8408eef98a0e5"
server_key_version: "encryptionkey@0"
I20260929 16:24:44.799821 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b"
format_stamp: "Formatted at 2026-09-29 16:24:44 on dist-test-slave-s9ct"
server_key: "f1981b293d9f507bd49c5be486bdd74a"
server_key_iv: "f921135bc8055fefb8f8408eef98a0e5"
server_key_version: "encryptionkey@0"
I20260929 16:24:44.803498 25714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:24:44.806067 25729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:44.807325 25714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:24:44.807475 25714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b"
format_stamp: "Formatted at 2026-09-29 16:24:44 on dist-test-slave-s9ct"
server_key: "f1981b293d9f507bd49c5be486bdd74a"
server_key_iv: "f921135bc8055fefb8f8408eef98a0e5"
server_key_version: "encryptionkey@0"
I20260929 16:24:44.807617 25714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:44.812105 25714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:44.815187 25714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:44.815397 25714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:44.822976 25714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40611
I20260929 16:24:44.823036 25781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40611 every 8 connection(s)
I20260929 16:24:44.823997 25714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:44.824283 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25714
I20260929 16:24:44.824424 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:44.824744 4897 external_mini_cluster.cc:1428] Setting key dbb2310317b57a51feb671ceac97fd60
I20260929 16:24:44.827170 25782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:44.830520 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:44.826338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47871 (local address 127.0.0.1:40611)
0929 16:24:44.826847 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:44.826867 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:44.826917 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:24:44.827684 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:44.827703 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:44.828004 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:44.828319 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:44.829200 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:44.829205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:44.829216 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:44.829228 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:44.829283 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:44.829302 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:44.829421 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:44.829530 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 16:24:44.829743 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":150,"threads_started":1}
I20260929 16:24:44.833107 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b: Bootstrap starting.
I20260929 16:24:44.833333 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:44.835815 25782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:44.836623 25782 log.cc:824] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:44.837301 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:44.835565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52365 (local address 127.0.0.1:40611)
0929 16:24:44.835711 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:44.835718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:44.835863 (+ 145us) server_negotiation.cc:407] Connection header received
0929 16:24:44.835971 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:44.835974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:44.836012 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:44.836107 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:44.836853 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:44.836857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:44.836859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:44.836871 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:44.836901 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:44.836915 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:44.837009 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:44.837134 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:24:44.837166 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:24:44.838793 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b: No bootstrap required, opened a new log
I20260929 16:24:44.842289 25782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260929 16:24:44.842564 25782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:44.842648 25782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4cfbfc290246f6a0ce9c4d71f0043b, State: Initialized, Role: FOLLOWER
I20260929 16:24:44.843063 25782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260929 16:24:44.843209 25782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:44.843283 25782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:44.843390 25782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:44.844287 25782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260929 16:24:44.844624 25782 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4cfbfc290246f6a0ce9c4d71f0043b; no voters:
I20260929 16:24:44.844936 25782 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:44.845114 25792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:44.845382 25792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [term 1 LEADER]: Becoming Leader. State: Replica: 3f4cfbfc290246f6a0ce9c4d71f0043b, State: Running, Role: LEADER
I20260929 16:24:44.845772 25792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260929 16:24:44.846177 25782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:44.847352 25793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } } }
I20260929 16:24:44.847468 25793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:44.847806 25794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4cfbfc290246f6a0ce9c4d71f0043b. Latest consensus state: current_term: 1 leader_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4cfbfc290246f6a0ce9c4d71f0043b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } } }
I20260929 16:24:44.847882 25794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:44.847870 25801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:44.851140 25801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:44.856834 25801 catalog_manager.cc:1347] Generated new cluster ID: 8fca9451707f49bb8f3e6707c1c007aa
I20260929 16:24:44.856912 25801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:44.880050 25801 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:44.880370 25801 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:44.887667 25801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4cfbfc290246f6a0ce9c4d71f0043b: Generated new TSK 0
I20260929 16:24:44.888440 25801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:44.901458 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25714
2026-09-29T16:24:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:44 dist-test-slave-s9ct krb5kdc[25813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:44 dist-test-slave-s9ct krb5kdc[25813](info): set up 2 sockets
Sep 29 16:24:44 dist-test-slave-s9ct krb5kdc[25813](info): commencing operation
krb5kdc: starting...
W20260929 16:24:45.938733 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:46.065385 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:46.178921 25826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:46.179261 25826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:46.179354 25826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:46.183463 25826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:46.183573 25826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:46.183609 25826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:46.183646 25826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:46.183677 25826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:46.189181 25826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:46.190325 25826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:46.191450 25826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:46.200385 25831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:46.200388 25834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:46.200389 25832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:46.200804 25826 server_base.cc:1034] running on GCE node
I20260929 16:24:46.201424 25826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:46.202675 25826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:46.203896 25826 hybrid_clock.cc:648] HybridClock initialized: now 1790699086203879 us; error 34 us; skew 500 ppm
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:46.215252 25826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:46.216288 25826 webserver.cc:460] Webserver started at http://127.0.0.1:44953/ using document root <none> and password file <none>
I20260929 16:24:46.216856 25826 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:46.216923 25826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:46.217096 25826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:46.219305 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7"
format_stamp: "Formatted at 2026-09-29 16:24:46 on dist-test-slave-s9ct"
server_key: "23fca1d718ccff4b12f7be07e8db280a"
server_key_iv: "f81b7d88555945d1f3e707cdca9725ea"
server_key_version: "encryptionkey@0"
I20260929 16:24:46.219848 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7"
format_stamp: "Formatted at 2026-09-29 16:24:46 on dist-test-slave-s9ct"
server_key: "23fca1d718ccff4b12f7be07e8db280a"
server_key_iv: "f81b7d88555945d1f3e707cdca9725ea"
server_key_version: "encryptionkey@0"
I20260929 16:24:46.223912 25826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:46.226630 25841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:46.227964 25826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:24:46.228157 25826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7"
format_stamp: "Formatted at 2026-09-29 16:24:46 on dist-test-slave-s9ct"
server_key: "23fca1d718ccff4b12f7be07e8db280a"
server_key_iv: "f81b7d88555945d1f3e707cdca9725ea"
server_key_version: "encryptionkey@0"
I20260929 16:24:46.228274 25826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:46.242367 25826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:46.245992 25826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:46.246251 25826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:46.253986 25826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43663
I20260929 16:24:46.254000 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43663 every 8 connection(s)
I20260929 16:24:46.255101 25826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:46.257944 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:46.262104 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25826
I20260929 16:24:46.262233 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:46.262606 4897 external_mini_cluster.cc:1428] Setting key 09d68bfd32e6d56138dd942dc2f10220
I20260929 16:24:46.263963 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: Bootstrap starting.
I20260929 16:24:46.266350 25894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:46.267261 25894 log.cc:824] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:46.269639 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: No bootstrap required, opened a new log
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:24:46.273991 25894 raft_consensus.cc:348] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260929 16:24:46.274250 25894 raft_consensus.cc:374] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:46.274343 25894 raft_consensus.cc:729] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b264dafa4ce5430d8e0a7c4c3cde3da7, State: Initialized, Role: FOLLOWER
I20260929 16:24:46.274823 25894 consensus_queue.cc:260] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260929 16:24:46.274983 25894 raft_consensus.cc:388] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:46.275074 25894 raft_consensus.cc:482] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:46.275195 25894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:46.276124 25894 raft_consensus.cc:504] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260929 16:24:46.276484 25894 leader_election.cc:302] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b264dafa4ce5430d8e0a7c4c3cde3da7; no voters:
I20260929 16:24:46.276826 25894 leader_election.cc:288] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:46.276942 25899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:46.277182 25899 raft_consensus.cc:686] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [term 1 LEADER]: Becoming Leader. State: Replica: b264dafa4ce5430d8e0a7c4c3cde3da7, State: Running, Role: LEADER
I20260929 16:24:46.277599 25899 consensus_queue.cc:237] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260929 16:24:46.278162 25894 sys_catalog.cc:564] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:46.278913 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b264dafa4ce5430d8e0a7c4c3cde3da7. Latest consensus state: current_term: 1 leader_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } } }
I20260929 16:24:46.279443 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:46.264216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:56171 (local address 127.0.0.1:43663)
0929 16:24:46.264638 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:46.264654 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:46.264689 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:24:46.265342 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:46.265359 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:46.265713 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:46.265995 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:46.267000 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:46.268360 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:46.269560 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:46.270198 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:46.270321 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:46.273271 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:46.273276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:46.273291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:46.273320 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:46.275449 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:46.276008 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:46.276013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:46.276022 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:46.276149 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:46.276504 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:46.276507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:46.276509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:46.277084 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:46.277925 (+ 841us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:46.278099 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:24:46.278339 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":136,"threads_started":1}
I20260929 16:24:46.279651 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:46.279541 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b264dafa4ce5430d8e0a7c4c3cde3da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } } }
I20260929 16:24:46.279847 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:46.283075 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 16:24:46.283537 25914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:46.283628 25914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:46.283710 25909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:46.284468 25909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:46.290293 25909 catalog_manager.cc:1347] Generated new cluster ID: 1bf6b8731abf4db1891e14488ba479e6
I20260929 16:24:46.290359 25909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:46.299258 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:46.286384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:33181 (local address 127.0.0.1:43663)
0929 16:24:46.286532 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:46.286538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:46.286838 (+ 300us) server_negotiation.cc:407] Connection header received
0929 16:24:46.286987 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:46.286990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:46.287038 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:46.287133 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:46.287809 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:46.288349 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:46.289299 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:46.289559 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:46.289679 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:46.295620 (+ 5941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:46.295624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:46.295626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:46.295654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:46.297351 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:46.297906 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:46.297909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:46.297911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:46.297989 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:46.298327 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:46.298330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:46.298332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:46.298555 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:46.298688 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:46.298997 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:24:46.299100 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:24:46.321759 25909 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:46.323184 25909 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:46.330361 25909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b264dafa4ce5430d8e0a7c4c3cde3da7: Generated new TSK 0
I20260929 16:24:46.331311 25909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:46.341157 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25826
W20260929 16:24:46.346539 25811 connection.cc:535] client connection to 127.0.0.1:43663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43663): Connection reset by peer (error 104)
2026-09-29T16:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25925](info): set up 2 sockets
Sep 29 16:24:46 dist-test-slave-s9ct krb5kdc[25925](info): commencing operation
krb5kdc: starting...
W20260929 16:24:47.378727 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:47 dist-test-slave-s9ct krb5kdc[25925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:47.506147 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:47.623656 25938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:47.623930 25938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:47.623974 25938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:47.628216 25938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:47.628296 25938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:47.628333 25938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:47.628376 25938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:47.628412 25938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:47.634039 25938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:47.635257 25938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:47.636349 25938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:47.645043 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:47.645104 25938 server_base.cc:1034] running on GCE node
W20260929 16:24:47.645051 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:47.645051 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:47.645922 25938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:47.647078 25938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:47.648263 25938 hybrid_clock.cc:648] HybridClock initialized: now 1790699087648242 us; error 31 us; skew 500 ppm
Sep 29 16:24:47 dist-test-slave-s9ct krb5kdc[25925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:47.650941 25938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:47.651746 25938 webserver.cc:460] Webserver started at http://127.0.0.1:33101/ using document root <none> and password file <none>
I20260929 16:24:47.652259 25938 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:47.652316 25938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:47.652478 25938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:47.654491 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "b4c1e1ddc6ff48f9884762739e02d44f"
format_stamp: "Formatted at 2026-09-29 16:24:47 on dist-test-slave-s9ct"
server_key: "15d3b32003d575d90f7e8142f72fd1e8"
server_key_iv: "549285d3cb3e3f0e15b5cb46ab24b13e"
server_key_version: "encryptionkey@0"
I20260929 16:24:47.654994 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "b4c1e1ddc6ff48f9884762739e02d44f"
format_stamp: "Formatted at 2026-09-29 16:24:47 on dist-test-slave-s9ct"
server_key: "15d3b32003d575d90f7e8142f72fd1e8"
server_key_iv: "549285d3cb3e3f0e15b5cb46ab24b13e"
server_key_version: "encryptionkey@0"
I20260929 16:24:47.658603 25938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:24:47.660908 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:47.662086 25938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:47.662180 25938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b4c1e1ddc6ff48f9884762739e02d44f"
format_stamp: "Formatted at 2026-09-29 16:24:47 on dist-test-slave-s9ct"
server_key: "15d3b32003d575d90f7e8142f72fd1e8"
server_key_iv: "549285d3cb3e3f0e15b5cb46ab24b13e"
server_key_version: "encryptionkey@0"
I20260929 16:24:47.662271 25938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:47.686659 25938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:47.689482 25938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:47.689651 25938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:47.696811 25938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34811
I20260929 16:24:47.696838 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34811 every 8 connection(s)
I20260929 16:24:47.697755 25938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:47.700416 26006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:47.702766 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 25938
I20260929 16:24:47.702913 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:47.703197 4897 external_mini_cluster.cc:1428] Setting key 3ff9990a29ff5ff32554ab68dd05fbc2
I20260929 16:24:47.706575 26006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f: Bootstrap starting.
I20260929 16:24:47.709071 26006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:47.709918 26006 log.cc:824] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:47.712015 26006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f: No bootstrap required, opened a new log
I20260929 16:24:47.713409 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:47.704645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:52967 (local address 127.0.0.1:34811)
0929 16:24:47.705059 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:47.705077 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:47.705127 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:24:47.705923 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:47.705946 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:47.706377 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:47.706748 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:47.707156 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:47.708620 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:47.709529 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:47.710240 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:47.710375 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:47.711401 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:47.711407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:47.711436 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:47.711936 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:47.712010 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:47.712043 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:47.712183 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:47.712388 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:24:47.712516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260929 16:24:47.715917 26006 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } }
I20260929 16:24:47.716164 26006 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:47.716272 26006 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1e1ddc6ff48f9884762739e02d44f, State: Initialized, Role: FOLLOWER
I20260929 16:24:47.716748 26006 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } }
I20260929 16:24:47.716856 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:47.716907 26006 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:47.716974 26006 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:47.717065 26006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:47.717934 26006 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } }
I20260929 16:24:47.718338 26006 leader_election.cc:302] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1e1ddc6ff48f9884762739e02d44f; no voters:
I20260929 16:24:47.718681 26006 leader_election.cc:288] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:47.718819 26015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:47.719050 26015 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [term 1 LEADER]: Becoming Leader. State: Replica: b4c1e1ddc6ff48f9884762739e02d44f, State: Running, Role: LEADER
I20260929 16:24:47.719494 26015 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } }
I20260929 16:24:47.720039 26006 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:47.721081 26018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c1e1ddc6ff48f9884762739e02d44f. Latest consensus state: current_term: 1 leader_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } } }
I20260929 16:24:47.721216 26018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:47.721506 26016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1e1ddc6ff48f9884762739e02d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34811 } } }
I20260929 16:24:47.721580 26016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:47.721601 26025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:47.724268 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:47.719264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:53745 (local address 127.0.0.1:34811)
0929 16:24:47.719357 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:47.719361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:47.719516 (+ 155us) server_negotiation.cc:407] Connection header received
0929 16:24:47.719698 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:47.719701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:47.719747 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:47.719796 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:47.720225 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:47.721010 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:47.722300 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:47.722684 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:47.722783 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:47.723756 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:47.723760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:47.723763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:47.723775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:47.723804 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:47.723819 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:47.723906 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:47.724082 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:24:47.724115 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 16:24:47.724778 26025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:47.729924 26025 catalog_manager.cc:1347] Generated new cluster ID: 5b9c80084795492b853cdc23dc9cf0b7
I20260929 16:24:47.729985 26025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:47.752399 26025 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:47.753328 26025 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:47.763052 26025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c1e1ddc6ff48f9884762739e02d44f: Generated new TSK 0
I20260929 16:24:47.763892 26025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:47.776420 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 25938
W20260929 16:24:47.781867 25923 connection.cc:535] client connection to 127.0.0.1:34811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34811): Connection reset by peer (error 104)
2026-09-29T16:24:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:47 dist-test-slave-s9ct krb5kdc[26037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:47 dist-test-slave-s9ct krb5kdc[26037](info): set up 2 sockets
Sep 29 16:24:47 dist-test-slave-s9ct krb5kdc[26037](info): commencing operation
krb5kdc: starting...
W20260929 16:24:48.814787 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:48 dist-test-slave-s9ct krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:48.942575 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:49.059583 26050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:49.059880 26050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:49.059924 26050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:49.064062 26050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:49.064137 26050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:49.064158 26050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:49.064175 26050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:49.064191 26050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:49.069716 26050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:49.070916 26050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:49.071951 26050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:49.081275 26055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:49.081278 26058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:49.081307 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:49.081570 26050 server_base.cc:1034] running on GCE node
I20260929 16:24:49.082376 26050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:49.083648 26050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:49.084851 26050 hybrid_clock.cc:648] HybridClock initialized: now 1790699089084835 us; error 37 us; skew 500 ppm
Sep 29 16:24:49 dist-test-slave-s9ct krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:49.088459 26050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:49.089510 26050 webserver.cc:460] Webserver started at http://127.0.0.1:45007/ using document root <none> and password file <none>
I20260929 16:24:49.090158 26050 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:49.090255 26050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:49.090539 26050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:49.092754 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "4ac95c3ce5f94e93af473238af929021"
format_stamp: "Formatted at 2026-09-29 16:24:49 on dist-test-slave-s9ct"
server_key: "e0254d1742d8b282ef62ddd9a42f6afe"
server_key_iv: "c06fa3597dba688465e3536d33de2a37"
server_key_version: "encryptionkey@0"
I20260929 16:24:49.093353 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "4ac95c3ce5f94e93af473238af929021"
format_stamp: "Formatted at 2026-09-29 16:24:49 on dist-test-slave-s9ct"
server_key: "e0254d1742d8b282ef62ddd9a42f6afe"
server_key_iv: "c06fa3597dba688465e3536d33de2a37"
server_key_version: "encryptionkey@0"
I20260929 16:24:49.097442 26050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:24:49.100234 26065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:49.101583 26050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:49.101737 26050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "4ac95c3ce5f94e93af473238af929021"
format_stamp: "Formatted at 2026-09-29 16:24:49 on dist-test-slave-s9ct"
server_key: "e0254d1742d8b282ef62ddd9a42f6afe"
server_key_iv: "c06fa3597dba688465e3536d33de2a37"
server_key_version: "encryptionkey@0"
I20260929 16:24:49.101866 26050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:49.106691 26050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:49.110098 26050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:49.110322 26050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:49.117834 26050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36989
I20260929 16:24:49.117851 26117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36989 every 8 connection(s)
I20260929 16:24:49.118897 26050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:49.121775 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:49.127689 26118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021: Bootstrap starting.
I20260929 16:24:49.128957 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26050
I20260929 16:24:49.129113 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:49.129411 4897 external_mini_cluster.cc:1428] Setting key ca0f673d68f298a8c548f7f38e0540d4
I20260929 16:24:49.130967 26118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:49.131811 26118 log.cc:824] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:49.133940 26118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021: No bootstrap required, opened a new log
I20260929 16:24:49.134930 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:49.130687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:47047 (local address 127.0.0.1:36989)
0929 16:24:49.131346 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:49.131361 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:49.131397 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:24:49.132153 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:49.132173 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:49.132470 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:49.132816 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:49.133644 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:49.133649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:49.133661 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:49.133671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:49.133725 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:49.133744 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:49.133863 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:49.133960 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 16:24:49.134132 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":137,"threads_started":1}
I20260929 16:24:49.137630 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:49.137892 26118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } }
I20260929 16:24:49.138146 26118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:49.138216 26118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac95c3ce5f94e93af473238af929021, State: Initialized, Role: FOLLOWER
I20260929 16:24:49.138700 26118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } }
I20260929 16:24:49.138860 26118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:49.138932 26118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:49.139038 26118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:49.140344 26118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } }
I20260929 16:24:49.140784 26118 leader_election.cc:302] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ac95c3ce5f94e93af473238af929021; no voters:
I20260929 16:24:49.141149 26118 leader_election.cc:288] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:49.141227 26128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:49.141479 26128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac95c3ce5f94e93af473238af929021, State: Running, Role: LEADER
I20260929 16:24:49.141794 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:49.139871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:58915 (local address 127.0.0.1:36989)
0929 16:24:49.139986 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:49.139995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:49.140242 (+ 247us) server_negotiation.cc:407] Connection header received
0929 16:24:49.140392 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:49.140394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:49.140424 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:49.140519 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:49.141346 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:49.141351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:49.141354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:49.141370 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:49.141401 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:49.141415 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:49.141475 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:49.141642 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:24:49.141683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:24:49.141824 26128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } }
I20260929 16:24:49.142642 26118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:49.143291 26128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ac95c3ce5f94e93af473238af929021. Latest consensus state: current_term: 1 leader_uuid: "4ac95c3ce5f94e93af473238af929021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } } }
I20260929 16:24:49.143455 26128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:49.143733 26129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ac95c3ce5f94e93af473238af929021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac95c3ce5f94e93af473238af929021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36989 } } }
I20260929 16:24:49.143890 26136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:49.143890 26129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:49.147209 26136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:49.152691 26136 catalog_manager.cc:1347] Generated new cluster ID: f01085a4355f43b792243110795e87b2
I20260929 16:24:49.152768 26136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:49.164412 26136 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:49.164732 26136 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:49.171868 26136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ac95c3ce5f94e93af473238af929021: Generated new TSK 0
I20260929 16:24:49.172650 26136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:49.182827 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26050
2026-09-29T16:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:49 dist-test-slave-s9ct krb5kdc[26149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:49 dist-test-slave-s9ct krb5kdc[26149](info): set up 2 sockets
Sep 29 16:24:49 dist-test-slave-s9ct krb5kdc[26149](info): commencing operation
krb5kdc: starting...
W20260929 16:24:50.214864 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:50.350297 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:50.462587 26162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:50.462883 26162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:50.462926 26162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:50.467070 26162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:50.467130 26162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:50.467149 26162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:50.467165 26162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:50.467202 26162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:50.472568 26162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:50.473661 26162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:50.474714 26162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:50.483636 26170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:50.483831 26162 server_base.cc:1034] running on GCE node
W20260929 16:24:50.483623 26167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:50.483623 26168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:50.484506 26162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:50.485598 26162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:50.486753 26162 hybrid_clock.cc:648] HybridClock initialized: now 1790699090486738 us; error 29 us; skew 500 ppm
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:50.489504 26162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:50.490386 26162 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
I20260929 16:24:50.490972 26162 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:50.491031 26162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:50.491195 26162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:50.493222 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "eb3dc789ad7c44549d7757c0d2b69277"
format_stamp: "Formatted at 2026-09-29 16:24:50 on dist-test-slave-s9ct"
server_key: "e199f47f826b57b2b540b24d0de5fa5b"
server_key_iv: "dfe821c8be8bcfa058b8cbd08cb8c6eb"
server_key_version: "encryptionkey@0"
I20260929 16:24:50.493737 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "eb3dc789ad7c44549d7757c0d2b69277"
format_stamp: "Formatted at 2026-09-29 16:24:50 on dist-test-slave-s9ct"
server_key: "e199f47f826b57b2b540b24d0de5fa5b"
server_key_iv: "dfe821c8be8bcfa058b8cbd08cb8c6eb"
server_key_version: "encryptionkey@0"
I20260929 16:24:50.497681 26162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:24:50.500177 26177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:50.501967 26162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:24:50.502156 26162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "eb3dc789ad7c44549d7757c0d2b69277"
format_stamp: "Formatted at 2026-09-29 16:24:50 on dist-test-slave-s9ct"
server_key: "e199f47f826b57b2b540b24d0de5fa5b"
server_key_iv: "dfe821c8be8bcfa058b8cbd08cb8c6eb"
server_key_version: "encryptionkey@0"
I20260929 16:24:50.502383 26162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:50.521435 26162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:50.524755 26162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:50.524964 26162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:50.531687 26162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39145
I20260929 16:24:50.531701 26229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39145 every 8 connection(s)
I20260929 16:24:50.532619 26162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:50.535395 26230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:50.537108 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26162
I20260929 16:24:50.537241 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:50.537516 4897 external_mini_cluster.cc:1428] Setting key cbb3de55a8417d989f6a986727cfd071
I20260929 16:24:50.541540 26230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: Bootstrap starting.
I20260929 16:24:50.543864 26230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:50.544677 26230 log.cc:824] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:24:50.546955 26230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: No bootstrap required, opened a new log
I20260929 16:24:50.550680 26230 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260929 16:24:50.550917 26230 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:50.551044 26230 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3dc789ad7c44549d7757c0d2b69277, State: Initialized, Role: FOLLOWER
I20260929 16:24:50.551566 26230 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260929 16:24:50.551731 26230 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:50.551826 26230 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:50.551937 26230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:50.552527 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:50.538889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:51725 (local address 127.0.0.1:39145)
0929 16:24:50.539333 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:50.539348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:50.539383 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:24:50.540019 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:50.540034 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:50.540386 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:50.540713 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:50.541529 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:50.542603 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:50.543470 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:50.543937 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:50.544108 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:50.547310 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:50.547316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:50.547331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:50.547362 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:50.549119 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:50.549659 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:50.549664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:50.549669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:50.549793 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:50.550126 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:50.550134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:50.550136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:50.550597 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:50.550827 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:50.551173 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:24:50.551422 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260929 16:24:50.553166 26230 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260929 16:24:50.553534 26230 leader_election.cc:302] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3dc789ad7c44549d7757c0d2b69277; no voters:
I20260929 16:24:50.553884 26230 leader_election.cc:288] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:50.554025 26235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:50.554234 26235 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [term 1 LEADER]: Becoming Leader. State: Replica: eb3dc789ad7c44549d7757c0d2b69277, State: Running, Role: LEADER
I20260929 16:24:50.554629 26235 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260929 16:24:50.555213 26230 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:50.555950 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:50.556250 26237 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3dc789ad7c44549d7757c0d2b69277. Latest consensus state: current_term: 1 leader_uuid: "eb3dc789ad7c44549d7757c0d2b69277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260929 16:24:50.556365 26237 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:50.556689 26236 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3dc789ad7c44549d7757c0d2b69277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3dc789ad7c44549d7757c0d2b69277" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260929 16:24:50.556759 26236 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277 [sys.catalog]: This master's current role is: LEADER
W20260929 16:24:50.560643 26250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:24:50.560739 26250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:24:50.560861 26244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:50.561550 26244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:50.567677 26244 catalog_manager.cc:1347] Generated new cluster ID: eb0fb0ceda224de282414c1a54cd6d32
I20260929 16:24:50.567726 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:50.558498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:33195 (local address 127.0.0.1:39145)
0929 16:24:50.558628 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:50.558635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:50.558971 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:24:50.559128 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:50.559131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:50.559184 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:50.559274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:50.560079 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:50.560849 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:50.562084 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:50.562482 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:50.562564 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:50.564270 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:50.564274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:50.564277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:50.564308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:50.565727 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:50.566280 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:50.566284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:50.566286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:50.566377 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:50.566745 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:50.566749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:50.566750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:50.566983 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:50.567136 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:50.567439 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:24:50.567545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:24:50.567751 26244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:50.597282 26244 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:50.598279 26244 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:50.605978 26244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3dc789ad7c44549d7757c0d2b69277: Generated new TSK 0
I20260929 16:24:50.606679 26244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:50.616680 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26162
W20260929 16:24:50.622207 26147 connection.cc:535] client connection to 127.0.0.1:39145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39145): Connection reset by peer (error 104)
2026-09-29T16:24:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26261](info): set up 2 sockets
Sep 29 16:24:50 dist-test-slave-s9ct krb5kdc[26261](info): commencing operation
krb5kdc: starting...
W20260929 16:24:51.639045 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:51 dist-test-slave-s9ct krb5kdc[26261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:51.759547 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:51.872475 26274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:51.872720 26274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:51.872761 26274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:51.876767 26274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:51.876823 26274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:51.876842 26274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:51.876858 26274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:51.876875 26274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:51.881906 26274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:51.882892 26274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:51.883833 26274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:51.891837 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:51.891772 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:51.891769 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:51.891976 26274 server_base.cc:1034] running on GCE node
I20260929 16:24:51.892632 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:51.893587 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:51.894730 26274 hybrid_clock.cc:648] HybridClock initialized: now 1790699091894717 us; error 24 us; skew 500 ppm
Sep 29 16:24:51 dist-test-slave-s9ct krb5kdc[26261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:51.905429 26274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:51.906313 26274 webserver.cc:460] Webserver started at http://127.0.0.1:41309/ using document root <none> and password file <none>
I20260929 16:24:51.906863 26274 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:51.906922 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:51.907133 26274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:51.909075 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "18f84d172ce142d78f16c80eeec2b69c"
format_stamp: "Formatted at 2026-09-29 16:24:51 on dist-test-slave-s9ct"
server_key: "40236b06e663cd9538728dd21fef35e5"
server_key_iv: "2ab1f6329e1c0c835f3d3f47af3cb2a1"
server_key_version: "encryptionkey@0"
I20260929 16:24:51.909611 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "18f84d172ce142d78f16c80eeec2b69c"
format_stamp: "Formatted at 2026-09-29 16:24:51 on dist-test-slave-s9ct"
server_key: "40236b06e663cd9538728dd21fef35e5"
server_key_iv: "2ab1f6329e1c0c835f3d3f47af3cb2a1"
server_key_version: "encryptionkey@0"
I20260929 16:24:51.913232 26274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:24:51.915499 26289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:51.916620 26274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:24:51.916736 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "18f84d172ce142d78f16c80eeec2b69c"
format_stamp: "Formatted at 2026-09-29 16:24:51 on dist-test-slave-s9ct"
server_key: "40236b06e663cd9538728dd21fef35e5"
server_key_iv: "2ab1f6329e1c0c835f3d3f47af3cb2a1"
server_key_version: "encryptionkey@0"
I20260929 16:24:51.916829 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:51.928040 26274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:51.931056 26274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:51.931237 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:51.938045 26274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44909
I20260929 16:24:51.938069 26341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44909 every 8 connection(s)
I20260929 16:24:51.938988 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:51.941545 26342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:51.945741 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26274
I20260929 16:24:51.945884 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:51.946134 4897 external_mini_cluster.cc:1428] Setting key 6a09412ccc49e7bf1258a7f835c51fcf
I20260929 16:24:51.946938 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c: Bootstrap starting.
I20260929 16:24:51.949157 26342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:51.949867 26342 log.cc:824] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:51.951618 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c: No bootstrap required, opened a new log
I20260929 16:24:51.954521 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:51.947459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:48831 (local address 127.0.0.1:44909)
0929 16:24:51.947857 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:51.947871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:51.947904 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:24:51.948489 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:51.948504 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:51.948821 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:51.949130 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:51.949517 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:51.950541 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:51.951402 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:51.951875 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:51.952039 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:51.952931 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:51.952936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:51.952947 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:51.953315 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:51.953365 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:51.953385 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:51.953516 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:51.953684 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:24:51.953810 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260929 16:24:51.955407 26342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260929 16:24:51.955668 26342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:51.955742 26342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f84d172ce142d78f16c80eeec2b69c, State: Initialized, Role: FOLLOWER
I20260929 16:24:51.956133 26342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260929 16:24:51.956275 26342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:51.956341 26342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:51.956455 26342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:51.957350 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:51.957315 26342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260929 16:24:51.957643 26342 leader_election.cc:302] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18f84d172ce142d78f16c80eeec2b69c; no voters:
I20260929 16:24:51.957957 26342 leader_election.cc:288] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:51.958041 26347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:51.958278 26347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [term 1 LEADER]: Becoming Leader. State: Replica: 18f84d172ce142d78f16c80eeec2b69c, State: Running, Role: LEADER
I20260929 16:24:51.958655 26347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } }
I20260929 16:24:51.959239 26342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:51.959735 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18f84d172ce142d78f16c80eeec2b69c. Latest consensus state: current_term: 1 leader_uuid: "18f84d172ce142d78f16c80eeec2b69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } } }
I20260929 16:24:51.959925 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:51.960219 26348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18f84d172ce142d78f16c80eeec2b69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18f84d172ce142d78f16c80eeec2b69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44909 } } }
I20260929 16:24:51.960318 26348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:51.960561 26359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:51.964025 26359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:51.966085 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:51.961477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:57861 (local address 127.0.0.1:44909)
0929 16:24:51.961572 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:51.961576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:51.961755 (+ 179us) server_negotiation.cc:407] Connection header received
0929 16:24:51.961902 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:51.961904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:51.961947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:51.962020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:51.962267 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:51.963038 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:51.964164 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:51.964501 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:51.964578 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:51.965594 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:51.965597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:51.965600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:51.965613 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:51.965639 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:51.965653 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:51.965746 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:51.965919 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:24:51.965951 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 16:24:51.969662 26359 catalog_manager.cc:1347] Generated new cluster ID: e22e1eaed35b42a0b16e96707cd95801
I20260929 16:24:51.969722 26359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:51.989360 26359 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:51.990207 26359 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:51.995206 26359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18f84d172ce142d78f16c80eeec2b69c: Generated new TSK 0
I20260929 16:24:51.995828 26359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:52.000623 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26274
W20260929 16:24:52.005983 26259 connection.cc:535] client connection to 127.0.0.1:44909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44909): Connection reset by peer (error 104)
2026-09-29T16:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:52 dist-test-slave-s9ct krb5kdc[26373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:52 dist-test-slave-s9ct krb5kdc[26373](info): set up 2 sockets
Sep 29 16:24:52 dist-test-slave-s9ct krb5kdc[26373](info): commencing operation
krb5kdc: starting...
W20260929 16:24:53.034750 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:53 dist-test-slave-s9ct krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:24:53.156927 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:53.265517 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:53.265766 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:53.265815 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:53.269670 26386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:53.269724 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:53.269743 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:53.269759 26386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:53.269774 26386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:53.274912 26386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:53.275903 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:53.276863 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:53.285264 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:53.285293 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:53.285272 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:53.285595 26386 server_base.cc:1034] running on GCE node
I20260929 16:24:53.286164 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:53.287283 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:53.288457 26386 hybrid_clock.cc:648] HybridClock initialized: now 1790699093288425 us; error 43 us; skew 500 ppm
Sep 29 16:24:53 dist-test-slave-s9ct krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:53.291258 26386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:24:53.292213 26386 webserver.cc:460] Webserver started at http://127.0.0.1:42479/ using document root <none> and password file <none>
I20260929 16:24:53.292742 26386 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:53.292809 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:53.292979 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:53.295096 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "b80fb67d1bb64050a176acede700d6c4"
format_stamp: "Formatted at 2026-09-29 16:24:53 on dist-test-slave-s9ct"
server_key: "55a65621b1e59303b9b186334ec42cf8"
server_key_iv: "1eef841430aaf2b7f3c911fa1d662d46"
server_key_version: "encryptionkey@0"
I20260929 16:24:53.295619 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "b80fb67d1bb64050a176acede700d6c4"
format_stamp: "Formatted at 2026-09-29 16:24:53 on dist-test-slave-s9ct"
server_key: "55a65621b1e59303b9b186334ec42cf8"
server_key_iv: "1eef841430aaf2b7f3c911fa1d662d46"
server_key_version: "encryptionkey@0"
I20260929 16:24:53.299563 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:24:53.301905 26401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:53.303179 26386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:53.303280 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "b80fb67d1bb64050a176acede700d6c4"
format_stamp: "Formatted at 2026-09-29 16:24:53 on dist-test-slave-s9ct"
server_key: "55a65621b1e59303b9b186334ec42cf8"
server_key_iv: "1eef841430aaf2b7f3c911fa1d662d46"
server_key_version: "encryptionkey@0"
I20260929 16:24:53.303364 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:53.307963 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:53.311210 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:53.311369 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:53.318487 26386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45803
I20260929 16:24:53.318548 26453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45803 every 8 connection(s)
I20260929 16:24:53.319535 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:53.322453 26454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:53.322929 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26386
I20260929 16:24:53.323064 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:53.323367 4897 external_mini_cluster.cc:1428] Setting key 7f8c7c0b9bcfb929939bac1964ee06d2
I20260929 16:24:53.328907 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:53.324717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:33197 (local address 127.0.0.1:45803)
0929 16:24:53.325144 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:53.325160 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:53.325196 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:24:53.325921 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:53.325940 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:53.326231 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:53.326594 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:53.327520 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:53.327526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:53.327539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:53.327551 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:53.327610 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:53.327634 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:53.327760 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:53.327926 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:24:53.328144 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":144,"threads_started":1}
I20260929 16:24:53.330214 26454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4: Bootstrap starting.
I20260929 16:24:53.331945 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:53.332827 26454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:53.333782 26454 log.cc:824] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:53.335959 26454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4: No bootstrap required, opened a new log
I20260929 16:24:53.336279 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:53.334504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:50393 (local address 127.0.0.1:45803)
0929 16:24:53.334645 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:53.334652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:53.334880 (+ 228us) server_negotiation.cc:407] Connection header received
0929 16:24:53.335037 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:53.335040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:53.335078 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:53.335170 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:53.335876 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:53.335880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:53.335882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:53.335894 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:53.335922 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:53.335939 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:53.336025 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:53.336096 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:24:53.336146 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:24:53.340044 26454 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260929 16:24:53.340317 26454 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:53.340399 26454 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80fb67d1bb64050a176acede700d6c4, State: Initialized, Role: FOLLOWER
I20260929 16:24:53.340874 26454 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260929 16:24:53.341009 26454 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:53.341102 26454 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:53.341223 26454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:53.342533 26454 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260929 16:24:53.342906 26454 leader_election.cc:302] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80fb67d1bb64050a176acede700d6c4; no voters:
I20260929 16:24:53.343290 26454 leader_election.cc:288] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:53.343410 26464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:53.343654 26464 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [term 1 LEADER]: Becoming Leader. State: Replica: b80fb67d1bb64050a176acede700d6c4, State: Running, Role: LEADER
I20260929 16:24:53.344038 26464 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260929 16:24:53.344686 26454 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:53.345790 26465 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80fb67d1bb64050a176acede700d6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
I20260929 16:24:53.345952 26465 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:53.346376 26473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:53.346344 26466 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80fb67d1bb64050a176acede700d6c4. Latest consensus state: current_term: 1 leader_uuid: "b80fb67d1bb64050a176acede700d6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fb67d1bb64050a176acede700d6c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
I20260929 16:24:53.347111 26466 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:53.349579 26473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:53.355371 26473 catalog_manager.cc:1347] Generated new cluster ID: 4cd5be2d1d1b41dc8b1aad488ebb308b
I20260929 16:24:53.355455 26473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:53.375931 26473 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:53.376317 26473 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:53.383118 26473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80fb67d1bb64050a176acede700d6c4: Generated new TSK 0
I20260929 16:24:53.383832 26473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:53.395308 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26386
2026-09-29T16:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:53 dist-test-slave-s9ct krb5kdc[26485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:53 dist-test-slave-s9ct krb5kdc[26485](info): set up 2 sockets
Sep 29 16:24:53 dist-test-slave-s9ct krb5kdc[26485](info): commencing operation
krb5kdc: starting...
W20260929 16:24:54.442735 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:54.567420 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40651
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:54.675336 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:54.675568 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:54.675608 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:54.679533 26498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:54.679582 26498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:54.679601 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:54.679617 26498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:54.679631 26498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:54.685935 26498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:40651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:54.686933 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:54.687886 26498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:54.696143 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:54.696134 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:54.696143 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:54.696395 26498 server_base.cc:1034] running on GCE node
I20260929 16:24:54.697098 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:54.698150 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:54.699325 26498 hybrid_clock.cc:648] HybridClock initialized: now 1790699094699304 us; error 34 us; skew 500 ppm
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699094, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:54.701881 26498 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:54.702767 26498 webserver.cc:460] Webserver started at http://127.4.200.126:46707/ using document root <none> and password file <none>
I20260929 16:24:54.703320 26498 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:54.703415 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:54.703639 26498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:54.705675 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "7835f0cb0f6c483c9212f13c334f3d75"
format_stamp: "Formatted at 2026-09-29 16:24:54 on dist-test-slave-s9ct"
server_key: "3cdb1865917abb23af4aad9f271f113a"
server_key_iv: "9aaf31d745e4c98110bdf6786ca62ddf"
server_key_version: "encryptionkey@0"
I20260929 16:24:54.706238 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "7835f0cb0f6c483c9212f13c334f3d75"
format_stamp: "Formatted at 2026-09-29 16:24:54 on dist-test-slave-s9ct"
server_key: "3cdb1865917abb23af4aad9f271f113a"
server_key_iv: "9aaf31d745e4c98110bdf6786ca62ddf"
server_key_version: "encryptionkey@0"
I20260929 16:24:54.709872 26498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:24:54.712311 26513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:54.713492 26498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:54.713625 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "7835f0cb0f6c483c9212f13c334f3d75"
format_stamp: "Formatted at 2026-09-29 16:24:54 on dist-test-slave-s9ct"
server_key: "3cdb1865917abb23af4aad9f271f113a"
server_key_iv: "9aaf31d745e4c98110bdf6786ca62ddf"
server_key_version: "encryptionkey@0"
I20260929 16:24:54.713727 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:54.734937 26498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:54.737679 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:54.737887 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:54.744688 26498 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:40651
I20260929 16:24:54.744702 26565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:40651 every 8 connection(s)
I20260929 16:24:54.745628 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:54.748332 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:54.753523 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26498
I20260929 16:24:54.753677 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:54.753790 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75: Bootstrap starting.
I20260929 16:24:54.753988 4897 external_mini_cluster.cc:1428] Setting key 16f1324fbb509109856087b50d353b10
I20260929 16:24:54.756281 26566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:54.757061 26566 log.cc:824] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:54.759007 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75: No bootstrap required, opened a new log
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:54.763291 26566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } }
I20260929 16:24:54.763522 26566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:54.763594 26566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7835f0cb0f6c483c9212f13c334f3d75, State: Initialized, Role: FOLLOWER
I20260929 16:24:54.764117 26566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } }
I20260929 16:24:54.764282 26566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:54.764367 26566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:54.764478 26566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:54.765317 26566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } }
I20260929 16:24:54.765647 26566 leader_election.cc:302] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7835f0cb0f6c483c9212f13c334f3d75; no voters:
I20260929 16:24:54.765928 26566 leader_election.cc:288] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:54.766026 26571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:54.766229 26571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [term 1 LEADER]: Becoming Leader. State: Replica: 7835f0cb0f6c483c9212f13c334f3d75, State: Running, Role: LEADER
I20260929 16:24:54.766594 26571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } }
I20260929 16:24:54.767122 26566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:54.768252 26573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7835f0cb0f6c483c9212f13c334f3d75. Latest consensus state: current_term: 1 leader_uuid: "7835f0cb0f6c483c9212f13c334f3d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } } }
I20260929 16:24:54.768378 26573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:54.768702 26572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7835f0cb0f6c483c9212f13c334f3d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7835f0cb0f6c483c9212f13c334f3d75" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40651 } } }
I20260929 16:24:54.768782 26572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75 [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:54.769057 26580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:54.769454 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:54.755251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60969 (local address 127.4.200.126:40651)
0929 16:24:54.755641 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:54.755663 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:54.755699 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:24:54.756310 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:54.756327 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:54.756695 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:54.756965 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:54.757831 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:54.758863 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:54.759992 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:54.760437 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:54.760548 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:54.763414 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:54.763419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:54.763431 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:54.763456 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:54.765215 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:54.765740 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:54.765746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:54.765751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:54.765916 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:54.766375 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:54.766378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:54.766380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:54.766893 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:54.767089 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:54.767979 (+ 890us) server_negotiation.cc:299] Negotiation successful
0929 16:24:54.768259 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260929 16:24:54.772313 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:54.772964 26580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:54.779045 26580 catalog_manager.cc:1347] Generated new cluster ID: 36728fa9f05042d39003a612ea023e04
I20260929 16:24:54.779111 26580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:54.782447 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:54.774494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57641 (local address 127.4.200.126:40651)
0929 16:24:54.774615 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:54.774620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:54.774819 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:24:54.774945 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:54.774948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:54.774992 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:54.775070 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:54.775691 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:54.776204 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:54.777153 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:54.777540 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:54.778021 (+ 481us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:54.779314 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:54.779318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:54.779321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:54.779344 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:54.780682 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:54.781143 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:54.781147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:54.781148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:54.781222 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:54.781548 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:54.781555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:54.781556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:54.781750 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:54.781927 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:54.782179 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:24:54.782267 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:24:54.794584 26580 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:54.795773 26580 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:54.802992 26580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7835f0cb0f6c483c9212f13c334f3d75: Generated new TSK 0
I20260929 16:24:54.803715 26580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:54.811453 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26498
W20260929 16:24:54.816716 26483 connection.cc:535] client connection to 127.4.200.126:40651 recv error: Network error: failed to read from TLS socket (remote: 127.4.200.126:40651): Connection reset by peer (error 104)
2026-09-29T16:24:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26597](info): set up 2 sockets
Sep 29 16:24:54 dist-test-slave-s9ct krb5kdc[26597](info): commencing operation
krb5kdc: starting...
W20260929 16:24:55.846800 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:55 dist-test-slave-s9ct krb5kdc[26597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:55.973630 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40747
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:56.083930 26610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:56.084214 26610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:56.084257 26610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:56.089561 26610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:56.089638 26610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:56.089660 26610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:56.089677 26610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:56.089694 26610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:56.095098 26610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:40747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:56.096084 26610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:56.097031 26610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:56.105690 26616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:56.105679 26615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:56.105777 26610 server_base.cc:1034] running on GCE node
W20260929 16:24:56.105681 26618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:56.106591 26610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:56.107694 26610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:56.108878 26610 hybrid_clock.cc:648] HybridClock initialized: now 1790699096108860 us; error 31 us; skew 500 ppm
Sep 29 16:24:56 dist-test-slave-s9ct krb5kdc[26597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:56.111630 26610 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:56.112535 26610 webserver.cc:460] Webserver started at http://127.4.200.126:35005/ using document root <none> and password file <none>
I20260929 16:24:56.113101 26610 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:56.113190 26610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:56.113426 26610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:56.115622 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c"
format_stamp: "Formatted at 2026-09-29 16:24:56 on dist-test-slave-s9ct"
server_key: "23bfc0e32888eac45e835d37917301f7"
server_key_iv: "336ac7d6c01dfda3851ac7450be378ef"
server_key_version: "encryptionkey@0"
I20260929 16:24:56.116205 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c"
format_stamp: "Formatted at 2026-09-29 16:24:56 on dist-test-slave-s9ct"
server_key: "23bfc0e32888eac45e835d37917301f7"
server_key_iv: "336ac7d6c01dfda3851ac7450be378ef"
server_key_version: "encryptionkey@0"
I20260929 16:24:56.119982 26610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:24:56.122360 26625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:56.123626 26610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:24:56.123773 26610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c"
format_stamp: "Formatted at 2026-09-29 16:24:56 on dist-test-slave-s9ct"
server_key: "23bfc0e32888eac45e835d37917301f7"
server_key_iv: "336ac7d6c01dfda3851ac7450be378ef"
server_key_version: "encryptionkey@0"
I20260929 16:24:56.123900 26610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:56.136963 26610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:56.139861 26610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:56.140053 26610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:56.147099 26610 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:40747
I20260929 16:24:56.147140 26677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:40747 every 8 connection(s)
I20260929 16:24:56.148028 26610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:56.150089 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26610
I20260929 16:24:56.150251 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:56.150578 4897 external_mini_cluster.cc:1428] Setting key 0995eac902a2c0ee74a9771dbb592bdd
I20260929 16:24:56.151269 26678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:56.157183 26678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c: Bootstrap starting.
I20260929 16:24:56.159518 26678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:56.159997 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:56.151963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56587 (local address 127.4.200.126:40747)
0929 16:24:56.152451 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:56.152469 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:56.152515 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:24:56.153164 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:56.153181 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:56.153517 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:56.153837 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:56.154233 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:56.155443 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:56.156450 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:56.156980 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:56.157170 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:56.158205 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:56.158211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:56.158223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:56.158691 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:56.158753 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:56.158778 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:56.158933 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:56.159102 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:24:56.159245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260929 16:24:56.160357 26678 log.cc:824] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:56.162472 26678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c: No bootstrap required, opened a new log
I20260929 16:24:56.163055 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:56.166757 26678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } }
I20260929 16:24:56.167088 26678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:56.167160 26678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054a7d08fbfa4f21b7eef1ee8e81ed6c, State: Initialized, Role: FOLLOWER
I20260929 16:24:56.167654 26678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } }
I20260929 16:24:56.167806 26678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:56.167897 26678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:56.168028 26678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:56.168998 26678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } }
I20260929 16:24:56.169349 26678 leader_election.cc:302] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054a7d08fbfa4f21b7eef1ee8e81ed6c; no voters:
I20260929 16:24:56.169593 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:56.165102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40179 (local address 127.4.200.126:40747)
0929 16:24:56.165219 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:56.165223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:56.165373 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:24:56.165515 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:56.165517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:56.165552 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:56.165617 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:56.165929 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:56.166477 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:56.167580 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:56.167823 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:56.167951 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:56.168943 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:56.168947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:56.168950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:56.168961 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:56.168993 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:56.169003 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:56.169098 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:56.169420 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:24:56.169455 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:24:56.169703 26678 leader_election.cc:288] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:56.170053 26688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:56.170244 26688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [term 1 LEADER]: Becoming Leader. State: Replica: 054a7d08fbfa4f21b7eef1ee8e81ed6c, State: Running, Role: LEADER
I20260929 16:24:56.170698 26688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } }
I20260929 16:24:56.171031 26678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:56.172406 26690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054a7d08fbfa4f21b7eef1ee8e81ed6c. Latest consensus state: current_term: 1 leader_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } } }
I20260929 16:24:56.172381 26689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054a7d08fbfa4f21b7eef1ee8e81ed6c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40747 } } }
I20260929 16:24:56.172537 26690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:56.172540 26689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:56.172974 26697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:56.176030 26697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:56.181777 26697 catalog_manager.cc:1347] Generated new cluster ID: 5f0e73be9fe346e8ab18c93239ba21e2
I20260929 16:24:56.181851 26697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:56.196110 26697 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:56.197062 26697 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:56.207336 26697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054a7d08fbfa4f21b7eef1ee8e81ed6c: Generated new TSK 0
I20260929 16:24:56.208254 26697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:56.216958 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26610
W20260929 16:24:56.222625 26595 connection.cc:535] client connection to 127.4.200.126:40747 recv error: Network error: failed to read from TLS socket (remote: 127.4.200.126:40747): Connection reset by peer (error 104)
2026-09-29T16:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:56 dist-test-slave-s9ct krb5kdc[26709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:56 dist-test-slave-s9ct krb5kdc[26709](info): set up 2 sockets
Sep 29 16:24:56 dist-test-slave-s9ct krb5kdc[26709](info): commencing operation
krb5kdc: starting...
W20260929 16:24:57.246879 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:57 dist-test-slave-s9ct krb5kdc[26709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:57.374271 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44799
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:57.481279 26722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:57.481576 26722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:57.481650 26722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:57.485607 26722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:57.485699 26722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:57.485739 26722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:57.485797 26722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:57.485843 26722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:57.491485 26722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:57.492527 26722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:57.493542 26722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:57.501940 26727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:57.501928 26728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:57.502014 26722 server_base.cc:1034] running on GCE node
W20260929 16:24:57.501909 26730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:57.502802 26722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:57.503854 26722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:57.505039 26722 hybrid_clock.cc:648] HybridClock initialized: now 1790699097505027 us; error 25 us; skew 500 ppm
Sep 29 16:24:57 dist-test-slave-s9ct krb5kdc[26709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:57.514592 26722 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:57.515956 26722 webserver.cc:460] Webserver started at http://127.4.200.126:35709/ using document root <none> and password file <none>
I20260929 16:24:57.516803 26722 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:57.516948 26722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:57.517266 26722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:57.520494 26722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "49f90a000d7548508caed2563b43f05d"
format_stamp: "Formatted at 2026-09-29 16:24:57 on dist-test-slave-s9ct"
server_key: "2abd627e683f77f7aa555f225b986b95"
server_key_iv: "b8ff38c49d68c438328f3fc81ac2db10"
server_key_version: "encryptionkey@0"
I20260929 16:24:57.521337 26722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "49f90a000d7548508caed2563b43f05d"
format_stamp: "Formatted at 2026-09-29 16:24:57 on dist-test-slave-s9ct"
server_key: "2abd627e683f77f7aa555f225b986b95"
server_key_iv: "b8ff38c49d68c438328f3fc81ac2db10"
server_key_version: "encryptionkey@0"
I20260929 16:24:57.526685 26722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:24:57.530040 26737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:57.531635 26722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:57.531821 26722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "49f90a000d7548508caed2563b43f05d"
format_stamp: "Formatted at 2026-09-29 16:24:57 on dist-test-slave-s9ct"
server_key: "2abd627e683f77f7aa555f225b986b95"
server_key_iv: "b8ff38c49d68c438328f3fc81ac2db10"
server_key_version: "encryptionkey@0"
I20260929 16:24:57.532032 26722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:57.539326 26722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:57.544234 26722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:57.544454 26722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:57.553673 26722 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:44799
I20260929 16:24:57.553777 26789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:44799 every 8 connection(s)
I20260929 16:24:57.554826 26722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:57.557725 26790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:57.560386 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26722
I20260929 16:24:57.560534 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:57.560830 4897 external_mini_cluster.cc:1428] Setting key 0097485442155ddd807f750871b241bf
I20260929 16:24:57.564113 26790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d: Bootstrap starting.
I20260929 16:24:57.566427 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:57.562130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35271 (local address 127.4.200.126:44799)
0929 16:24:57.562567 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:57.562586 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:57.562721 (+ 135us) server_negotiation.cc:407] Connection header received
0929 16:24:57.563560 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:57.563582 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:57.563943 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:57.564307 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:57.565015 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:57.565021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:57.565035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:57.565046 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:57.565106 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:57.565132 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:57.565287 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:57.565377 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 16:24:57.565539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":101,"threads_started":1}
I20260929 16:24:57.566847 26790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:57.567648 26790 log.cc:824] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:57.569237 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:57.569883 26790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d: No bootstrap required, opened a new log
I20260929 16:24:57.572898 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:57.571286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51691 (local address 127.4.200.126:44799)
0929 16:24:57.571405 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:57.571409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:57.571649 (+ 240us) server_negotiation.cc:407] Connection header received
0929 16:24:57.571763 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:57.571765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:57.571802 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:57.571865 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:57.572523 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:57.572525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:57.572528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:24:57.572537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:57.572558 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:24:57.572568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:57.572633 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:57.572747 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:24:57.572787 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:24:57.574095 26790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } }
I20260929 16:24:57.574335 26790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:57.574441 26790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f90a000d7548508caed2563b43f05d, State: Initialized, Role: FOLLOWER
I20260929 16:24:57.574944 26790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } }
I20260929 16:24:57.575152 26790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:57.575239 26790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:57.575364 26790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:57.576292 26790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } }
I20260929 16:24:57.576643 26790 leader_election.cc:302] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f90a000d7548508caed2563b43f05d; no voters:
I20260929 16:24:57.576953 26790 leader_election.cc:288] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:57.577065 26800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:57.577315 26800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [term 1 LEADER]: Becoming Leader. State: Replica: 49f90a000d7548508caed2563b43f05d, State: Running, Role: LEADER
I20260929 16:24:57.577594 26800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } }
I20260929 16:24:57.578104 26790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:57.579207 26801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f90a000d7548508caed2563b43f05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } } }
I20260929 16:24:57.579332 26801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:57.579612 26802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f90a000d7548508caed2563b43f05d. Latest consensus state: current_term: 1 leader_uuid: "49f90a000d7548508caed2563b43f05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f90a000d7548508caed2563b43f05d" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44799 } } }
I20260929 16:24:57.579713 26802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:57.579793 26809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:57.582934 26809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:57.589169 26809 catalog_manager.cc:1347] Generated new cluster ID: 2b911bcb74114d5ead2079517e8d32c3
I20260929 16:24:57.589247 26809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:57.606048 26809 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:57.606467 26809 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:57.610603 26809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f90a000d7548508caed2563b43f05d: Generated new TSK 0
I20260929 16:24:57.611236 26809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:57.615592 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26722
2026-09-29T16:24:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:57 dist-test-slave-s9ct krb5kdc[26821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:57 dist-test-slave-s9ct krb5kdc[26821](info): set up 2 sockets
Sep 29 16:24:57 dist-test-slave-s9ct krb5kdc[26821](info): commencing operation
krb5kdc: starting...
W20260929 16:24:58.651047 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:24:58 dist-test-slave-s9ct krb5kdc[26821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:24:58.774375 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40773
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:39615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:24:58.888466 26834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:24:58.888758 26834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:24:58.888803 26834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:24:58.892791 26834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:24:58.892853 26834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:24:58.892872 26834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:24:58.892889 26834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:24:58.892904 26834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:24:58.898075 26834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:39615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:40773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:24:58.899164 26834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:24:58.900122 26834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:24:58.908550 26839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:24:58.908567 26840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:58.908620 26834 server_base.cc:1034] running on GCE node
W20260929 16:24:58.908545 26842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:24:58.909330 26834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:24:58.910583 26834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:24:58.911747 26834 hybrid_clock.cc:648] HybridClock initialized: now 1790699098911735 us; error 26 us; skew 500 ppm
Sep 29 16:24:58 dist-test-slave-s9ct krb5kdc[26821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:24:58.914615 26834 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:24:58.915545 26834 webserver.cc:460] Webserver started at http://127.4.200.126:41105/ using document root <none> and password file <none>
I20260929 16:24:58.916136 26834 fs_manager.cc:359] Metadata directory not provided
I20260929 16:24:58.916229 26834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:24:58.916473 26834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:24:58.918623 26834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "f0749d7f32654fb9acc957cf8afbb48e"
format_stamp: "Formatted at 2026-09-29 16:24:58 on dist-test-slave-s9ct"
server_key: "cee554748f356a9b77f6bb3f4cd29ef2"
server_key_iv: "053bf2e877f69bf899d4d7502c6ffa68"
server_key_version: "encryptionkey@0"
I20260929 16:24:58.919188 26834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "f0749d7f32654fb9acc957cf8afbb48e"
format_stamp: "Formatted at 2026-09-29 16:24:58 on dist-test-slave-s9ct"
server_key: "cee554748f356a9b77f6bb3f4cd29ef2"
server_key_iv: "053bf2e877f69bf899d4d7502c6ffa68"
server_key_version: "encryptionkey@0"
I20260929 16:24:58.922988 26834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:24:58.925343 26849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:24:58.926877 26834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:24:58.927011 26834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "f0749d7f32654fb9acc957cf8afbb48e"
format_stamp: "Formatted at 2026-09-29 16:24:58 on dist-test-slave-s9ct"
server_key: "cee554748f356a9b77f6bb3f4cd29ef2"
server_key_iv: "053bf2e877f69bf899d4d7502c6ffa68"
server_key_version: "encryptionkey@0"
I20260929 16:24:58.927140 26834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:24:58.949783 26834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:24:58.953176 26834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:24:58.953416 26834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:24:58.960920 26834 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:40773
I20260929 16:24:58.960924 26901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:40773 every 8 connection(s)
I20260929 16:24:58.961922 26834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:24:58.964696 26902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:24:58.970496 26902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e: Bootstrap starting.
I20260929 16:24:58.971562 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26834
I20260929 16:24:58.971729 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:24:58.972276 4897 external_mini_cluster.cc:1428] Setting key e4cf7e5ea51f40b15ddc911566f8b4d8
I20260929 16:24:58.973856 26902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e: Neither blocks nor log segments found. Creating new log.
I20260929 16:24:58.974682 26902 log.cc:824] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:24:58.976918 26902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e: No bootstrap required, opened a new log
Sep 29 16:24:58 dist-test-slave-s9ct krb5kdc[26821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:24:58.981956 26902 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } }
I20260929 16:24:58.982275 26902 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:24:58.982363 26902 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0749d7f32654fb9acc957cf8afbb48e, State: Initialized, Role: FOLLOWER
I20260929 16:24:58.982890 26902 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } }
I20260929 16:24:58.983048 26902 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:24:58.983150 26902 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:24:58.983291 26902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:24:58.984356 26902 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } }
I20260929 16:24:58.984738 26902 leader_election.cc:302] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0749d7f32654fb9acc957cf8afbb48e; no voters:
I20260929 16:24:58.985145 26902 leader_election.cc:288] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:24:58.985232 26907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:24:58.985469 26907 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [term 1 LEADER]: Becoming Leader. State: Replica: f0749d7f32654fb9acc957cf8afbb48e, State: Running, Role: LEADER
I20260929 16:24:58.985816 26907 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } }
I20260929 16:24:58.986645 26902 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:24:58.987572 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0749d7f32654fb9acc957cf8afbb48e. Latest consensus state: current_term: 1 leader_uuid: "f0749d7f32654fb9acc957cf8afbb48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } } }
I20260929 16:24:58.987744 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:58.988143 26916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:24:58.988368 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:58.973446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51165 (local address 127.4.200.126:40773)
0929 16:24:58.973868 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:58.973883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:58.973943 (+ 60us) server_negotiation.cc:407] Connection header received
0929 16:24:58.974711 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:58.974730 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:58.975189 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:58.975555 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:58.976433 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:58.977480 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:58.978840 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:58.979308 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:58.979432 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:58.982619 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:58.982624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:58.982639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:58.982693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:58.984666 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:58.985199 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:58.985214 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:58.985223 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:58.985402 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:58.985779 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:58.985782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:58.985785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:58.986259 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:58.986453 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:58.987011 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:24:58.987323 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":142,"threads_started":1}
I20260929 16:24:58.989516 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0749d7f32654fb9acc957cf8afbb48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0749d7f32654fb9acc957cf8afbb48e" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 40773 } } }
I20260929 16:24:58.989698 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e [sys.catalog]: This master's current role is: LEADER
I20260929 16:24:58.991779 26916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:24:58.991771 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:24:59.000517 26916 catalog_manager.cc:1347] Generated new cluster ID: 52f918651721439c977c08090f1e24e3
I20260929 16:24:59.000593 26916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:24:59.012455 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:24:58.994097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37341 (local address 127.4.200.126:40773)
0929 16:24:58.994202 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:24:58.994206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:24:58.994577 (+ 371us) server_negotiation.cc:407] Connection header received
0929 16:24:58.994759 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:24:58.994763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:24:58.994811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:24:58.994874 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:24:58.995818 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:58.996603 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:58.997504 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:24:58.997773 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:24:58.997890 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:24:58.999342 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:24:58.999346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:24:58.999349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:24:58.999376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:24:59.001006 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:59.010947 (+ 9941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:59.010951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:59.010953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:59.011068 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:24:59.011479 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:24:59.011483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:24:59.011485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:24:59.011712 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:24:59.011847 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:24:59.012145 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:24:59.012276 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:24:59.015758 26916 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:24:59.016731 26916 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:24:59.028898 26916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0749d7f32654fb9acc957cf8afbb48e: Generated new TSK 0
I20260929 16:24:59.029850 26916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:24:59.040196 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26834
2026-09-29T16:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:24:59 dist-test-slave-s9ct krb5kdc[26933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:24:59 dist-test-slave-s9ct krb5kdc[26933](info): set up 2 sockets
Sep 29 16:24:59 dist-test-slave-s9ct krb5kdc[26933](info): commencing operation
krb5kdc: starting...
W20260929 16:25:00.114841 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:25:00 dist-test-slave-s9ct krb5kdc[26933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:25:00.242046 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44239
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:00.375779 26946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:00.376071 26946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:00.376158 26946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:00.380347 26946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:00.380409 26946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:00.380431 26946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:00.380448 26946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:00.380463 26946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:00.385946 26946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:00.387084 26946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:00.388118 26946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:00.396588 26951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:00.396588 26954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:00.396749 26946 server_base.cc:1034] running on GCE node
W20260929 16:25:00.396929 26952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:00.397511 26946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:00.398747 26946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:00.399935 26946 hybrid_clock.cc:648] HybridClock initialized: now 1790699100399908 us; error 42 us; skew 500 ppm
Sep 29 16:25:00 dist-test-slave-s9ct krb5kdc[26933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699100, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:00.402683 26946 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:25:00.403544 26946 webserver.cc:460] Webserver started at http://127.4.200.126:35605/ using document root <none> and password file <none>
I20260929 16:25:00.404106 26946 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:00.404197 26946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:00.404426 26946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:00.406517 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "acc2dcfd1c1d47f782faf5ae7402090c"
format_stamp: "Formatted at 2026-09-29 16:25:00 on dist-test-slave-s9ct"
server_key: "935596068f7f4f35b32119bf1ee4f861"
server_key_iv: "89fd250a9b869150db1af4fc87ddb1b0"
server_key_version: "encryptionkey@0"
I20260929 16:25:00.407080 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "acc2dcfd1c1d47f782faf5ae7402090c"
format_stamp: "Formatted at 2026-09-29 16:25:00 on dist-test-slave-s9ct"
server_key: "935596068f7f4f35b32119bf1ee4f861"
server_key_iv: "89fd250a9b869150db1af4fc87ddb1b0"
server_key_version: "encryptionkey@0"
I20260929 16:25:00.410828 26946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:25:00.413281 26961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:00.414716 26946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:00.414872 26946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "acc2dcfd1c1d47f782faf5ae7402090c"
format_stamp: "Formatted at 2026-09-29 16:25:00 on dist-test-slave-s9ct"
server_key: "935596068f7f4f35b32119bf1ee4f861"
server_key_iv: "89fd250a9b869150db1af4fc87ddb1b0"
server_key_version: "encryptionkey@0"
I20260929 16:25:00.414996 26946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:00.435593 26946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:00.438848 26946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:00.439059 26946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:00.445856 26946 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:44239
I20260929 16:25:00.445914 27013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:44239 every 8 connection(s)
I20260929 16:25:00.446862 26946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:00.448792 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 26946
I20260929 16:25:00.448943 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:25:00.449220 4897 external_mini_cluster.cc:1428] Setting key b97fbc2ca555651f990b339534ced24b
I20260929 16:25:00.449831 27014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:00.456885 27014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c: Bootstrap starting.
I20260929 16:25:00.459285 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:00.450620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47909 (local address 127.4.200.126:44239)
0929 16:25:00.451039 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:00.451058 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:00.451100 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:25:00.451968 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:00.451991 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:00.452428 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:00.452792 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:00.453210 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:00.454682 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:00.455585 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:00.456250 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:00.456431 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:00.457285 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:00.457292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:00.457308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:25:00.457781 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:00.457851 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:25:00.457882 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:00.457979 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:00.458083 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 16:25:00.458325 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260929 16:25:00.459717 27014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:00.460512 27014 log.cc:824] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:00.462363 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:25:00.462839 27014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c: No bootstrap required, opened a new log
I20260929 16:25:00.466658 27014 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } }
I20260929 16:25:00.466913 27014 raft_consensus.cc:374] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:00.467005 27014 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc2dcfd1c1d47f782faf5ae7402090c, State: Initialized, Role: FOLLOWER
I20260929 16:25:00.467500 27014 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } }
I20260929 16:25:00.467679 27014 raft_consensus.cc:388] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:25:00.467768 27014 raft_consensus.cc:482] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:25:00.467871 27014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:00.468351 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:00.464413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50067 (local address 127.4.200.126:44239)
0929 16:25:00.464528 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:00.464533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:00.464703 (+ 170us) server_negotiation.cc:407] Connection header received
0929 16:25:00.464866 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:00.464868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:00.464906 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:00.464987 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:00.465250 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:00.465781 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:00.466667 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:00.466909 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:00.467039 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:00.467878 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:00.467882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:00.467884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:25:00.467895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:00.467919 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:25:00.467929 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:00.468008 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:00.468149 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 16:25:00.468215 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:25:00.469079 27014 raft_consensus.cc:504] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } }
I20260929 16:25:00.469432 27014 leader_election.cc:302] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acc2dcfd1c1d47f782faf5ae7402090c; no voters:
I20260929 16:25:00.469727 27014 leader_election.cc:288] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:25:00.469853 27024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:00.470109 27024 raft_consensus.cc:686] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [term 1 LEADER]: Becoming Leader. State: Replica: acc2dcfd1c1d47f782faf5ae7402090c, State: Running, Role: LEADER
I20260929 16:25:00.470465 27024 consensus_queue.cc:237] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } }
I20260929 16:25:00.471026 27014 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:00.472214 27025 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } } }
I20260929 16:25:00.472357 27025 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:00.472223 27026 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc2dcfd1c1d47f782faf5ae7402090c. Latest consensus state: current_term: 1 leader_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc2dcfd1c1d47f782faf5ae7402090c" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44239 } } }
I20260929 16:25:00.472484 27026 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:00.473284 27034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:00.475970 27034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:00.481293 27034 catalog_manager.cc:1347] Generated new cluster ID: 469160018a114aa39faa49862d5223aa
I20260929 16:25:00.481359 27034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:00.489976 27034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:00.490988 27034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:00.500380 27034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acc2dcfd1c1d47f782faf5ae7402090c: Generated new TSK 0
I20260929 16:25:00.501353 27034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:25:00.507678 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 26946
2026-09-29T16:25:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:00 dist-test-slave-s9ct krb5kdc[27045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:00 dist-test-slave-s9ct krb5kdc[27045](info): set up 2 sockets
Sep 29 16:25:00 dist-test-slave-s9ct krb5kdc[27045](info): commencing operation
krb5kdc: starting...
W20260929 16:25:01.538987 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:25:01 dist-test-slave-s9ct krb5kdc[27045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:25:01.665632 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44791
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:01.775930 27058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:01.776245 27058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:01.776320 27058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:01.780437 27058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:01.780534 27058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:01.780572 27058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:01.780632 27058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:01.780671 27058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:01.786023 27058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:01.787226 27058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:01.788275 27058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:01.796856 27066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:01.796861 27063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:01.796934 27058 server_base.cc:1034] running on GCE node
W20260929 16:25:01.797024 27064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:01.797663 27058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:01.798921 27058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:01.800112 27058 hybrid_clock.cc:648] HybridClock initialized: now 1790699101800111 us; error 45 us; skew 500 ppm
Sep 29 16:25:01 dist-test-slave-s9ct krb5kdc[27045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:01.802989 27058 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:25:01.803954 27058 webserver.cc:460] Webserver started at http://127.4.200.126:45175/ using document root <none> and password file <none>
I20260929 16:25:01.804535 27058 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:01.804627 27058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:01.804877 27058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:01.807092 27058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "9a4311d1565b4db894079d0a4e4852a4"
format_stamp: "Formatted at 2026-09-29 16:25:01 on dist-test-slave-s9ct"
server_key: "1aeb757c0d1fff1c101a2fadf6e1e5dc"
server_key_iv: "bcb66aa5a03e288743037ae9c89a3fe8"
server_key_version: "encryptionkey@0"
I20260929 16:25:01.807677 27058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "9a4311d1565b4db894079d0a4e4852a4"
format_stamp: "Formatted at 2026-09-29 16:25:01 on dist-test-slave-s9ct"
server_key: "1aeb757c0d1fff1c101a2fadf6e1e5dc"
server_key_iv: "bcb66aa5a03e288743037ae9c89a3fe8"
server_key_version: "encryptionkey@0"
I20260929 16:25:01.811386 27058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:25:01.813848 27073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:01.815122 27058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:25:01.815267 27058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "9a4311d1565b4db894079d0a4e4852a4"
format_stamp: "Formatted at 2026-09-29 16:25:01 on dist-test-slave-s9ct"
server_key: "1aeb757c0d1fff1c101a2fadf6e1e5dc"
server_key_iv: "bcb66aa5a03e288743037ae9c89a3fe8"
server_key_version: "encryptionkey@0"
I20260929 16:25:01.815388 27058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:01.820107 27058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:01.823408 27058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:01.823611 27058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:01.831105 27058 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:44791
I20260929 16:25:01.831128 27125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:44791 every 8 connection(s)
I20260929 16:25:01.832084 27058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:01.835214 27126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:01.841218 27126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: Bootstrap starting.
I20260929 16:25:01.842208 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27058
I20260929 16:25:01.842356 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:25:01.842681 4897 external_mini_cluster.cc:1428] Setting key 30c15f562735d5363a300587dccbcff6
I20260929 16:25:01.844288 27126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:01.845141 27126 log.cc:824] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:01.847731 27126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: No bootstrap required, opened a new log
I20260929 16:25:01.848351 27129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:01.843832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38675 (local address 127.4.200.126:44791)
0929 16:25:01.844341 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:01.844355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:01.844394 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:25:01.845140 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:01.845159 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:01.845543 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:01.845985 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:01.846930 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:01.846934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:01.846946 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:25:01.846957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:01.847008 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:25:01.847027 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:01.847115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:01.847269 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:25:01.847441 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":174,"threads_started":1}
I20260929 16:25:01.851612 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:25:01.852468 27126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } }
I20260929 16:25:01.852801 27126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:01.852880 27126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4311d1565b4db894079d0a4e4852a4, State: Initialized, Role: FOLLOWER
I20260929 16:25:01.853418 27126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } }
I20260929 16:25:01.853609 27126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:25:01.853688 27126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:25:01.853798 27126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:01.855343 27126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } }
I20260929 16:25:01.855752 27126 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a4311d1565b4db894079d0a4e4852a4; no voters:
I20260929 16:25:01.855907 27129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:01.854115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37027 (local address 127.4.200.126:44791)
0929 16:25:01.854251 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:01.854256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:01.854452 (+ 196us) server_negotiation.cc:407] Connection header received
0929 16:25:01.854597 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:01.854599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:01.854631 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:01.854696 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:01.855484 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:01.855490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:01.855495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:25:01.855508 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:01.855544 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:25:01.855555 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:01.855624 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:01.855735 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:25:01.855766 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:25:01.856196 27126 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:25:01.856266 27136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:01.856513 27136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4311d1565b4db894079d0a4e4852a4, State: Running, Role: LEADER
I20260929 16:25:01.856889 27136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } }
I20260929 16:25:01.857622 27126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:01.858834 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4311d1565b4db894079d0a4e4852a4. Latest consensus state: current_term: 1 leader_uuid: "9a4311d1565b4db894079d0a4e4852a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } } }
I20260929 16:25:01.858876 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a4311d1565b4db894079d0a4e4852a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4311d1565b4db894079d0a4e4852a4" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44791 } } }
I20260929 16:25:01.859028 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:01.859035 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4 [sys.catalog]: This master's current role is: LEADER
W20260929 16:25:01.862845 27152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:01.862922 27152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:01.862999 27147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:01.863719 27147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:01.868517 27147 catalog_manager.cc:1347] Generated new cluster ID: 51baf688005a4f26a72ae5312d9a3c64
I20260929 16:25:01.868582 27147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:01.881625 27147 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:01.881928 27147 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:01.885241 27147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4311d1565b4db894079d0a4e4852a4: Generated new TSK 0
I20260929 16:25:01.885854 27147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:25:01.893657 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27058
2026-09-29T16:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1385 ms)
[----------] 18 tests from AuthTokenIssuingTest (25516 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:01 dist-test-slave-s9ct krb5kdc[27157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:01 dist-test-slave-s9ct krb5kdc[27157](info): set up 2 sockets
Sep 29 16:25:01 dist-test-slave-s9ct krb5kdc[27157](info): commencing operation
krb5kdc: starting...
W20260929 16:25:02.922902 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:25:03.048955 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44705
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:03.161643 27170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:25:03.162017 27170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:03.162096 27170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:03.162173 27170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:03.166313 27170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:03.166443 27170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:03.166493 27170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:03.166528 27170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:03.166558 27170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:03.171901 27170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:03.172986 27170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:03.174000 27170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:03.182350 27178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:03.182402 27176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:03.182454 27170 server_base.cc:1034] running on GCE node
W20260929 16:25:03.182350 27175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:03.183175 27170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:03.184278 27170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:03.185449 27170 hybrid_clock.cc:648] HybridClock initialized: now 1790699103185424 us; error 36 us; skew 500 ppm
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:03.195538 27170 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:25:03.196512 27170 webserver.cc:460] Webserver started at http://127.4.200.126:40589/ using document root <none> and password file <none>
I20260929 16:25:03.197093 27170 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:03.197156 27170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:03.197386 27170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:03.199545 27170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "bc347704995447a3b8d8d57d6bf84fc1"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d345d377375959652a6eb339c724a715"
server_key_iv: "bcff913ac26bccf284b413d8aea072ef"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.200103 27170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "bc347704995447a3b8d8d57d6bf84fc1"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d345d377375959652a6eb339c724a715"
server_key_iv: "bcff913ac26bccf284b413d8aea072ef"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.203877 27170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:25:03.206382 27185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:03.207670 27170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:25:03.207810 27170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "bc347704995447a3b8d8d57d6bf84fc1"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d345d377375959652a6eb339c724a715"
server_key_iv: "bcff913ac26bccf284b413d8aea072ef"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.207929 27170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:03.222229 27170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:03.225219 27170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:03.225430 27170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:03.232739 27170 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:44705
I20260929 16:25:03.232995 27237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:44705 every 8 connection(s)
I20260929 16:25:03.233822 27170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:03.235469 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27170
I20260929 16:25:03.235668 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:25:03.236001 4897 external_mini_cluster.cc:1428] Setting key f96ff95d1d73734f00449913ed0e8d3f
I20260929 16:25:03.236851 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:03.240568 27238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.200.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.125@KRBTEST.COM" created.
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:03.256150 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.242812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:44705 (local address 127.0.0.1:48780)
0929 16:25:03.243284 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.243335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.244289 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.244980 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.244984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.245304 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.245623 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.245635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.247100 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.247106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.248163 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.248173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.249046 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.249048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.249124 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.249732 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.249747 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.251185 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.253110 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.253120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.253130 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.253362 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.253684 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.253687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.253689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.253801 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.254379 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.254385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.254547 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.254721 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.255131 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":104,"thread_start_us":131,"threads_started":1}
I20260929 16:25:03.256150 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.243691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48780 (local address 127.4.200.126:44705)
0929 16:25:03.243972 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.243982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.244020 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:25:03.244615 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.244619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.244794 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.244951 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.245758 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.246962 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.248285 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.248914 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.249034 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.251350 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.251356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.251366 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.251385 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.253002 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.253470 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.253482 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.253485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.253580 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.253897 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.253900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.253902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.254265 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.254428 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.254795 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.255082 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260929 16:25:03.258622 27238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.259407 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.259147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38262)
0929 16:25:03.259263 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.259341 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:25:03.259698 27188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
I20260929 16:25:03.260833 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.260573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38268)
0929 16:25:03.260687 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.260733 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:25:03.260978 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:43963: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
I20260929 16:25:03.263545 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:43963
--webserver_interface=127.4.200.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:25:03.310266 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } attempt: 1
I20260929 16:25:03.311220 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.310922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38282)
0929 16:25:03.311074 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.311142 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:25:03.312026 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.311820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38290)
0929 16:25:03.311922 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.311967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:25:03.312235 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:43963: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
W20260929 16:25:03.376901 27245 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:25:03.377218 27245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:03.377242 27245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:03.377329 27245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:25:03.380419 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } attempt: 2
I20260929 16:25:03.381275 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.380981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38306)
0929 16:25:03.381129 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.381193 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:25:03.381927 27245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:03.382001 27245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:03.382064 27245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260929 16:25:03.382054 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.381805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38320)
0929 16:25:03.381913 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.381966 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:25:03.382124 27245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:03.382157 27245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:25:03.382277 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:43963: Network error: Client connection negotiation failed: client connection to 127.4.200.125:43963: connect: Connection refused (error 111)
I20260929 16:25:03.387640 27245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.200.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:03.388715 27245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:03.389758 27245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:25:03.398343 27245 server_base.cc:1034] running on GCE node
W20260929 16:25:03.398293 27251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:03.398314 27253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:03.398293 27250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:03.399190 27245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:03.400291 27245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:03.401435 27245 hybrid_clock.cc:648] HybridClock initialized: now 1790699103401418 us; error 30 us; skew 500 ppm
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:03.404204 27245 init.cc:377] Logged in from keytab as kudu/127.4.200.125@KRBTEST.COM (short username kudu)
I20260929 16:25:03.405051 27245 webserver.cc:460] Webserver started at http://127.4.200.125:33005/ using document root <none> and password file <none>
I20260929 16:25:03.405608 27245 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:03.405678 27245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:03.405959 27245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:03.408023 27245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data/instance:
uuid: "ea4cd9eb95f04482ab2662d04f3e0555"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "6063850f12c83f5c0e06203787d080b4"
server_key_iv: "ccf475669e60433acfec60ef31c2d489"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.408581 27245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal/instance:
uuid: "ea4cd9eb95f04482ab2662d04f3e0555"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "6063850f12c83f5c0e06203787d080b4"
server_key_iv: "ccf475669e60433acfec60ef31c2d489"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.412534 27245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:25:03.415076 27260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:03.416301 27245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:25:03.416440 27245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "ea4cd9eb95f04482ab2662d04f3e0555"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "6063850f12c83f5c0e06203787d080b4"
server_key_iv: "ccf475669e60433acfec60ef31c2d489"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.416548 27245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:03.431711 27245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:03.434911 27245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:03.435075 27245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:03.442107 27245 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.125:43963
I20260929 16:25:03.442462 27312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.125:43963 every 8 connection(s)
I20260929 16:25:03.443192 27245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
I20260929 16:25:03.446151 27313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:03.450114 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27245
I20260929 16:25:03.450245 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-1/wal/instance
I20260929 16:25:03.449899 27313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.450564 4897 external_mini_cluster.cc:1428] Setting key 4a49af2538e21576242c0a1dadfaaa9e
WARNING: no policy specified for kudu/127.4.200.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.124@KRBTEST.COM" created.
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:03.467536 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.452257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.4.200.126:44705)
0929 16:25:03.452457 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.452465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.453354 (+ 889us) server_negotiation.cc:407] Connection header received
0929 16:25:03.454349 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.454355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.454455 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.454569 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.455812 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.456726 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.458140 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.458577 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.458707 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.463005 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.463008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.463011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.463040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.464955 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.465558 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.465565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.465570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.465672 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.466040 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.466046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.466051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.466357 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.466484 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.467116 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.467277 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:25:03.468236 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.452643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:44705 (local address 127.0.0.1:48786)
0929 16:25:03.453166 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.453229 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.454111 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.454764 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.454771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.455128 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.455626 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.455640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.456924 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.456928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.457993 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.458004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.458782 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.458785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.458958 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.459549 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.459568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.462741 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.465106 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.465112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.465128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.465438 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.465788 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.465790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.465791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.465935 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.466523 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.466529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.466830 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.467006 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.467287 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":157,"threads_started":1}
I20260929 16:25:03.469846 27313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.473436 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } attempt: 3
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
I20260929 16:25:03.479236 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:34211
--webserver_interface=127.4.200.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
I20260929 16:25:03.479894 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.470446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38328)
0929 16:25:03.470546 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.470562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.470642 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.471168 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.471171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.471343 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.471431 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.471437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.472394 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.472400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.472943 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.472951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.473568 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.473571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.473672 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.474197 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.474217 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.475607 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.477856 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.477860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.477866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.478175 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.478537 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.478543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.478548 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.478650 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.479218 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.479222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.479331 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.479452 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.479607 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 16:25:03.480600 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.470534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.4.200.125:43963)
0929 16:25:03.470836 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.470848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.470865 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:25:03.470897 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.470901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.471052 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.471194 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.471538 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.472271 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.473057 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.473453 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.473568 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.475752 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.475763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.475784 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.475804 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.477743 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.478305 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.478311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.478313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.478435 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.478754 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.478757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.478759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.479093 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.479237 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.480148 (+ 911us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.480314 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260929 16:25:03.482957 27313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.483767 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.483469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:48936)
0929 16:25:03.483587 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.483689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:25:03.483982 27261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.484836 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.484601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:48952)
0929 16:25:03.484715 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.484771 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:25:03.485085 27313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.486024 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.474060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.4.200.125:43963)
0929 16:25:03.474536 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.474549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.474565 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:25:03.474603 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.474606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.474652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.474748 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.475219 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.475920 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.476920 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.477238 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.477325 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.479811 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.479815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.479817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.479840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.483166 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.483694 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.483698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.483700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.483785 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.485046 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.485049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.485051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.485301 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.485397 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.485782 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.485887 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260929 16:25:03.486143 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.474025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38344)
0929 16:25:03.474236 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.474272 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.474357 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.474770 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.474773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.474992 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.475105 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.475111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.476076 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.476081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.476796 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.476809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.477362 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.477366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.477503 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.478016 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.478032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.479672 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.483283 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.483290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.483295 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.483582 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.484800 (+ 1218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.484815 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.484818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.484943 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.485415 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.485421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.485562 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.485700 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.485897 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 16:25:03.487073 27238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.487783 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.487562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:48968)
0929 16:25:03.487678 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.487724 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:25:03.488487 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.488242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:48984)
0929 16:25:03.488390 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.488439 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 16:25:03.488633 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.519801 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 1
I20260929 16:25:03.520754 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.520452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:48996)
0929 16:25:03.520605 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.520673 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:25:03.521579 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.521344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49006)
0929 16:25:03.521450 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.521503 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:25:03.521741 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.534350 27313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 1
I20260929 16:25:03.535346 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.535006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49010)
0929 16:25:03.535179 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.535259 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:25:03.536106 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.535920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49016)
0929 16:25:03.536011 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.536054 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 16:25:03.536248 27313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.596920 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 2
W20260929 16:25:03.597569 27320 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260929 16:25:03.597884 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.597547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49032)
0929 16:25:03.597716 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.597782 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:25:03.597972 27320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:03.598034 27320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:03.598093 27320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:25:03.598661 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.598470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49048)
0929 16:25:03.598567 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.598610 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 16:25:03.598788 27238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
W20260929 16:25:03.602620 27320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:03.602696 27320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:03.602741 27320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:03.602775 27320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:03.602810 27320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:03.604413 27313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 2
I20260929 16:25:03.605209 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.604909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49060)
0929 16:25:03.605063 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.605121 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:25:03.605997 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.605779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49070)
0929 16:25:03.605889 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.605942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:25:03.606127 27313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:34211: Network error: Client connection negotiation failed: client connection to 127.4.200.124:34211: connect: Connection refused (error 111)
I20260929 16:25:03.608808 27320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:44705,127.4.200.125:43963,127.4.200.124:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.200.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:03.609920 27320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:03.611001 27320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:03.619534 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:03.619534 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:03.619674 27327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:03.619912 27320 server_base.cc:1034] running on GCE node
I20260929 16:25:03.620473 27320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:03.621579 27320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:03.622761 27320 hybrid_clock.cc:648] HybridClock initialized: now 1790699103622749 us; error 26 us; skew 500 ppm
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:03.625495 27320 init.cc:377] Logged in from keytab as kudu/127.4.200.124@KRBTEST.COM (short username kudu)
I20260929 16:25:03.626454 27320 webserver.cc:460] Webserver started at http://127.4.200.124:34507/ using document root <none> and password file <none>
I20260929 16:25:03.626995 27320 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:03.627054 27320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:03.627221 27320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:03.629230 27320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data/instance:
uuid: "a91daa7736554efa90da93e88c625771"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d8f4b00cb5e7eadab765d3998577f4d9"
server_key_iv: "00dd2ecf46daade2ace2dd775080b451"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.629751 27320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal/instance:
uuid: "a91daa7736554efa90da93e88c625771"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d8f4b00cb5e7eadab765d3998577f4d9"
server_key_iv: "00dd2ecf46daade2ace2dd775080b451"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.633826 27320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:25:03.636439 27336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:03.638062 27320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:25:03.638192 27320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "a91daa7736554efa90da93e88c625771"
format_stamp: "Formatted at 2026-09-29 16:25:03 on dist-test-slave-s9ct"
server_key: "d8f4b00cb5e7eadab765d3998577f4d9"
server_key_iv: "00dd2ecf46daade2ace2dd775080b451"
server_key_version: "encryptionkey@0"
I20260929 16:25:03.638307 27320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:03.684752 27320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:03.688390 27320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:03.688582 27320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:03.696583 27320 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.124:34211
I20260929 16:25:03.696969 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.124:34211 every 8 connection(s)
I20260929 16:25:03.697306 27313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 3
I20260929 16:25:03.697763 27320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
I20260929 16:25:03.697963 27238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } attempt: 3
I20260929 16:25:03.698318 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27320
I20260929 16:25:03.698477 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790698821455595-4897-0/minicluster-data/master-2/wal/instance
I20260929 16:25:03.698773 4897 external_mini_cluster.cc:1428] Setting key f2de9a269fcdc0f09d4ff9b3af5ddef3
I20260929 16:25:03.701591 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:03.706600 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:25:03.714618 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.697896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49084)
0929 16:25:03.698054 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.698072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.698211 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.701543 (+ 3332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.701546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.701795 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.701942 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.701952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.704396 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.704400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.705294 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.705307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.706493 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.706498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.706591 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.707346 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.707361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.708928 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.712427 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.712430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.712433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.712700 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.713088 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.713091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.713093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.713184 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.714076 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.714079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.714176 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.714268 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.714447 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:25:03.715335 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.699054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49100)
0929 16:25:03.699521 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.699538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.699630 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.701540 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.701544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.701774 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.701908 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.701916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.704386 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.704390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.705190 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.705204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.706439 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.706443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.706535 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.707630 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.707649 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.710024 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.713706 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.713709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.713711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.713950 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.714254 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.714257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.714260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.714344 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.714801 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.714804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.714891 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.714987 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.715119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367}
I20260929 16:25:03.716398 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.699210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.4.200.124:34211)
0929 16:25:03.699820 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.699841 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.699885 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:25:03.700701 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.700721 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.701188 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.702130 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.702670 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.704239 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.705471 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.706313 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.706884 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.709151 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.709158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.709178 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.709215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.712287 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.712811 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.712818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.712827 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.712969 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.713293 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.713299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.713303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.713950 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.715008 (+ 1058us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.715070 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.715370 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1295,"server-negotiator.queue_time_us":452,"spinlock_wait_cycles":49024,"thread_start_us":142,"threads_started":1}
I20260929 16:25:03.716395 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.699837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.4.200.124:34211)
0929 16:25:03.700121 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.700133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.700147 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:25:03.700699 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.700721 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.701188 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.702129 (+ 941us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.702629 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.704205 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.705366 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.706271 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.708263 (+ 1992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.710191 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.710194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.710197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.710220 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.713587 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.714052 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.714055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.714056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.714139 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.714460 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.714463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.714465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.714662 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.714754 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.715655 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.715791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":217,"spinlock_wait_cycles":30592,"thread_start_us":121,"threads_started":1}
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:03.716398 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.700094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.4.200.124:34211)
0929 16:25:03.700325 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.700344 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.700354 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:25:03.700701 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.700721 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.701188 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.702464 (+ 1276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.702647 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.704239 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.705471 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.706271 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.706893 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.710250 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.710254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.710257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.710285 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.712545 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.713059 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.713065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.713069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.713149 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.713472 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.713475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.713476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.713950 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.714710 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.714934 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.715313 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":559,"server-negotiator.queue_time_us":172,"spinlock_wait_cycles":71552,"thread_start_us":74,"threads_started":1}
I20260929 16:25:03.720929 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.708782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.4.200.126:44705)
0929 16:25:03.708912 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.708916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.709327 (+ 411us) server_negotiation.cc:407] Connection header received
0929 16:25:03.709499 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.709504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.709555 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.709618 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.712191 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.712928 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.714157 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.714513 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.714582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.717545 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.717548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.717551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.717575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.719299 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.719698 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.719701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.719703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.719783 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.720016 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.720021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.720022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.720277 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.720382 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.720684 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.720795 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:25:03.720944 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.708735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:44705 (local address 127.0.0.1:48790)
0929 16:25:03.709212 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.709261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.709408 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.709884 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.709888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.710318 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.712069 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.712082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.713030 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.713036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.714037 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.714047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.714996 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.714999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.715111 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.715885 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.715898 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.717372 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.719396 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.719403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.719407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.719608 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.719862 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.719864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.719866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.719932 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.720385 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.720389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.720480 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.720614 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.720759 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260929 16:25:03.721374 27238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: Bootstrap starting.
I20260929 16:25:03.722358 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.724647 27238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:03.725554 27313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: Bootstrap starting.
I20260929 16:25:03.725543 27238 log.cc:824] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:03.727612 27238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: No bootstrap required, opened a new log
I20260929 16:25:03.728250 27313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:03.729027 27313 log.cc:824] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:03.729641 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.719032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.4.200.126:44705)
0929 16:25:03.721088 (+ 2056us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.721092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.721105 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:25:03.721134 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.721136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.721187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.721265 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.721976 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.722636 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.723601 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.723827 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.723933 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.726371 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.726375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.726378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.726452 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.727971 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.728410 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.728413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.728414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.728481 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.728756 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.728759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.728760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.728929 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.729183 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.729283 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.729386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2002,"thread_start_us":82,"threads_started":1}
I20260929 16:25:03.731817 27238 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.732028 27238 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc347704995447a3b8d8d57d6bf84fc1, State: Initialized, Role: FOLLOWER
I20260929 16:25:03.732354 27238 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.732853 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.733062 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.733798 27238 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
I20260929 16:25:03.734920 27313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: No bootstrap required, opened a new log
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
I20260929 16:25:03.739050 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.723036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:43963 (local address 127.0.0.1:38346)
0929 16:25:03.729489 (+ 6453us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.729504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.729627 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.729971 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.729974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.730140 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.730263 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.730270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.731334 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.731337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.732106 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.732116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.732821 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.732824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.732920 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.733397 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.733412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.735092 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.737171 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.737176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.737178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.737423 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.737768 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.737774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.737777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.737884 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.738455 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.738461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.738570 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.738685 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.738852 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6376}
I20260929 16:25:03.739322 27313 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.739542 27313 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4cd9eb95f04482ab2662d04f3e0555, State: Initialized, Role: FOLLOWER
I20260929 16:25:03.739738 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.723291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.4.200.125:43963)
0929 16:25:03.726518 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.726523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.729564 (+ 3041us) server_negotiation.cc:407] Connection header received
0929 16:25:03.729737 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.729741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.729790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.729840 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.730421 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.731191 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.732249 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.732594 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.732676 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.735242 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.735248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.735254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.735284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.737027 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.737580 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.737584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.737586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.737667 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.737992 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.737996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.738000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.738288 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.738386 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.739456 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.739575 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3150}
I20260929 16:25:03.739856 27313 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.740527 27411 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.740726 27411 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.741230 27313 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:03.741359 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.730760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.4.200.125:43963)
0929 16:25:03.730874 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.730879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.730913 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:25:03.731040 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.731043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.731078 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.731166 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.731721 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.732318 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.733644 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.734115 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.734188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.737658 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.737661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.737664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.737683 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.739413 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.739882 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.739887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.739890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.739975 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.740279 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.740282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.740284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.740526 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.740685 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.741056 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.741164 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:25:03.742959 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:03.738780 27170 master_runner.cc:186] Error getting master registration for 127.4.200.125:43963: OK, instance_id { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" instance_seqno: 1790699103440158 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:25:03.744068 27170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:25:03.745113 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:25:03.746467 27212 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc347704995447a3b8d8d57d6bf84fc1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48806
I20260929 16:25:03.746670 27212 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:25:03.746790 27212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:25:03.747229 27410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:03.747294 27410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:03.748090 27212 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
Sep 29 16:25:03 dist-test-slave-s9ct krb5kdc[27157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:25:03.749434 27287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc347704995447a3b8d8d57d6bf84fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ea4cd9eb95f04482ab2662d04f3e0555"
I20260929 16:25:03.749635 27287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:03.750905 27287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc347704995447a3b8d8d57d6bf84fc1 in term 1.
I20260929 16:25:03.751320 27188 leader_election.cc:302] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc347704995447a3b8d8d57d6bf84fc1, ea4cd9eb95f04482ab2662d04f3e0555; no voters:
I20260929 16:25:03.751611 27399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:03.751820 27399 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [term 1 LEADER]: Becoming Leader. State: Replica: bc347704995447a3b8d8d57d6bf84fc1, State: Running, Role: LEADER
I20260929 16:25:03.752151 27399 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.753142 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.743468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:34211 (local address 127.0.0.1:49120)
0929 16:25:03.743837 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:03.743851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:03.743926 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:03.744486 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:03.744491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:03.744669 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:03.744749 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.744754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.745866 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.745868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.746608 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:03.746617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.747200 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.747204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:03.747296 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.747789 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:03.747803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:03.749314 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:03.751410 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.751413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.751415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.751667 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.752079 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:03.752081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:03.752083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:03.752174 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:03.752669 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:03.752672 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:03.752754 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 16:25:03.752850 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:25:03.752980 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260929 16:25:03.753705 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.744079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.4.200.124:34211)
0929 16:25:03.744204 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.744210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.744226 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:25:03.744258 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.744260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.744299 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.744381 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.744869 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.745758 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.746731 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.747076 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.747180 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.749460 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.749464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.749466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.749490 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.751236 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.751852 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.751856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.751858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.751985 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.752266 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.752269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.752271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.752549 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.752637 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.753465 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.753575 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:25:03.753769 27422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:03.753842 27422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:03.753993 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc347704995447a3b8d8d57d6bf84fc1. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.754067 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:03.754840 27212 leader_election.cc:288] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [CANDIDATE]: Term 1 election: Requested vote from peers ea4cd9eb95f04482ab2662d04f3e0555 (127.4.200.125:43963), a91daa7736554efa90da93e88c625771 (127.4.200.124:34211)
I20260929 16:25:03.755225 27427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:03.756138 27427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:03.757117 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: Bootstrap starting.
I20260929 16:25:03.760444 27389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:03.760697 27245 master_runner.cc:186] Error getting master registration for 127.4.200.124:34211: OK, instance_id { permanent_uuid: "a91daa7736554efa90da93e88c625771" instance_seqno: 1790699103694192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:25:03.764974 27245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:25:03.765908 27389 log.cc:824] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:03.775743 27287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [term 1 FOLLOWER]: Refusing update from remote peer bc347704995447a3b8d8d57d6bf84fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:25:03.776503 27424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:25:03.782151 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.763589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.4.200.125:43963)
0929 16:25:03.763947 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.763953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.770673 (+ 6720us) server_negotiation.cc:407] Connection header received
0929 16:25:03.770798 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.770802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.770857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.770941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.772257 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.773058 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.774759 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.775073 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.775153 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.777740 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.777743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.777757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.777784 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.779651 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.780425 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.780429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.780431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.780515 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.780996 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.781002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.781007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.781216 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.781313 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.781868 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.781996 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260929 16:25:03.782475 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: No bootstrap required, opened a new log
I20260929 16:25:03.786509 27411 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc347704995447a3b8d8d57d6bf84fc1. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.786652 27411 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.787508 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.763421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.4.200.124:34211)
0929 16:25:03.763598 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.763603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.763617 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:25:03.763658 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.763660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.763702 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.763777 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.765150 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.765895 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.767240 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.767593 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.767686 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.769427 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.769430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.769433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.769458 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.771177 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.773422 (+ 2245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.773425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.773427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.773522 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.774711 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.774717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.774722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.774945 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.775046 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.787217 (+ 12171us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.787349 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 16:25:03.794596 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:03.763225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.4.200.126:44705)
0929 16:25:03.763336 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.763340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.770982 (+ 7642us) server_negotiation.cc:407] Connection header received
0929 16:25:03.771152 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.771155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.771211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.771287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.772297 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.773057 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.787142 (+ 14085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:03.787495 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:03.787613 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:03.790038 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:03.790042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:03.790045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:03.790075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:03.791796 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.792601 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.792605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.792607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.792699 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:03.793248 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:03.793251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:03.793253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:03.793509 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:03.793630 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:03.794263 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:25:03.794427 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 16:25:03.794482 27389 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.794780 27389 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91daa7736554efa90da93e88c625771, State: Initialized, Role: FOLLOWER
I20260929 16:25:03.795281 27389 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } }
I20260929 16:25:03.796909 27389 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:03.797178 27362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:03.798880 27362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [term 1 FOLLOWER]: Refusing update from remote peer bc347704995447a3b8d8d57d6bf84fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:25:03.799443 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc347704995447a3b8d8d57d6bf84fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a91daa7736554efa90da93e88c625771"
I20260929 16:25:03.799552 27424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:25:03.800755 27363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc347704995447a3b8d8d57d6bf84fc1 in term 1.
I20260929 16:25:03.801708 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.801833 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:03.801955 27432 mvcc.cc:205] Tried to move back new op lower bound from 7334703529055490048 to 7334703528972472320. Current Snapshot: MvccSnapshot[applied={T|T < 7334703529055490048}]
I20260929 16:25:03.802277 27411 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.802563 27411 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.802989 27437 mvcc.cc:205] Tried to move back new op lower bound from 7334703529055490048 to 7334703528972472320. Current Snapshot: MvccSnapshot[applied={T|T < 7334703529055490048}]
I20260929 16:25:03.805531 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.805661 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.805757 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc347704995447a3b8d8d57d6bf84fc1. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.805840 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.807983 27427 catalog_manager.cc:1347] Generated new cluster ID: 1c7f9139744c4112b2a045fe27753234
I20260929 16:25:03.808054 27427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260929 16:25:03.814214 27455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:03.822439 27455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:03.824728 27424 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.824870 27424 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:03.826052 27448 mvcc.cc:205] Tried to move back new op lower bound from 7334703529055490048 to 7334703528972472320. Current Snapshot: MvccSnapshot[applied={T|T < 7334703529055490048}]
I20260929 16:25:03.826236 27447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc347704995447a3b8d8d57d6bf84fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc347704995447a3b8d8d57d6bf84fc1" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 44705 } } peers { permanent_uuid: "ea4cd9eb95f04482ab2662d04f3e0555" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 43963 } } peers { permanent_uuid: "a91daa7736554efa90da93e88c625771" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 34211 } } }
I20260929 16:25:03.826354 27447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:03.869303 27427 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:03.870635 27427 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:03.884065 27427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc347704995447a3b8d8d57d6bf84fc1: Generated new TSK 0
I20260929 16:25:03.885044 27427 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:25:03.931246 27394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.927989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.4.200.124:34211)
0929 16:25:03.928116 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.928121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.930094 (+ 1973us) server_negotiation.cc:407] Connection header received
0929 16:25:03.930306 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.930313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.930365 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.930447 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.931103 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49132: BlockingRecv error: recv got EOF from 127.0.0.1:49132 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:25:03.932683 27242 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.928447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.4.200.126:44705)
0929 16:25:03.928560 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.928564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.931741 (+ 3177us) server_negotiation.cc:407] Connection header received
0929 16:25:03.931918 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.931921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.931971 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.932020 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.932555 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48818: BlockingRecv error: recv got EOF from 127.0.0.1:48818 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 16:25:03.934168 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.928876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.4.200.125:43963)
0929 16:25:03.928993 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.928997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.933304 (+ 4307us) server_negotiation.cc:407] Connection header received
0929 16:25:03.933476 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.933479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.933528 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.933578 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.934049 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38384: BlockingRecv error: recv got EOF from 127.0.0.1:38384 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:25:03.948274 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.941289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.4.200.125:43963)
0929 16:25:03.941428 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.941433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.947018 (+ 5585us) server_negotiation.cc:407] Connection header received
0929 16:25:03.947300 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.947305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.947396 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.947465 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.948146 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38398: BlockingRecv error: recv got EOF from 127.0.0.1:38398 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 16:25:03.954531 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.952533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.4.200.125:43963)
0929 16:25:03.952673 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.952677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.953366 (+ 689us) server_negotiation.cc:407] Connection header received
0929 16:25:03.953606 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.953611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.953692 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.953753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.954370 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38400: BlockingRecv error: recv got EOF from 127.0.0.1:38400 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:25:03.964673 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.962910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.4.200.125:43963)
0929 16:25:03.963048 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.963053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.963668 (+ 615us) server_negotiation.cc:407] Connection header received
0929 16:25:03.963881 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.963885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.963940 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.963993 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.964534 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38414: BlockingRecv error: recv got EOF from 127.0.0.1:38414 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:25:03.973423 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.971564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.4.200.125:43963)
0929 16:25:03.971770 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.971776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.972423 (+ 647us) server_negotiation.cc:407] Connection header received
0929 16:25:03.972634 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.972640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.972707 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.972766 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.973300 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38426: BlockingRecv error: recv got EOF from 127.0.0.1:38426 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260929 16:25:03.985293 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:03.984002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.4.200.125:43963)
0929 16:25:03.984129 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:03.984134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:03.984391 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:25:03.984561 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:03.984564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:03.984622 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:03.984713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:03.985176 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38432: BlockingRecv error: recv got EOF from 127.0.0.1:38432 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:25:04.004597 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:04.003328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.4.200.125:43963)
0929 16:25:04.003498 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:04.003503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:04.003667 (+ 164us) server_negotiation.cc:407] Connection header received
0929 16:25:04.003817 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:04.003820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:04.003887 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:04.003962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:04.004468 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38442: BlockingRecv error: recv got EOF from 127.0.0.1:38442 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 16:25:04.039980 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:04.038912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.4.200.125:43963)
0929 16:25:04.039085 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:04.039088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:04.039151 (+ 63us) server_negotiation.cc:407] Connection header received
0929 16:25:04.039274 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:04.039277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:04.039333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:04.039406 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:04.039843 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38456: BlockingRecv error: recv got EOF from 127.0.0.1:38456 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:25:04.107810 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:04.106521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.4.200.125:43963)
0929 16:25:04.106711 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:04.106717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:04.106879 (+ 162us) server_negotiation.cc:407] Connection header received
0929 16:25:04.107055 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:04.107059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:04.107117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:04.107210 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:04.107687 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38458: BlockingRecv error: recv got EOF from 127.0.0.1:38458 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 16:25:04.239215 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:04.237978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.4.200.125:43963)
0929 16:25:04.238136 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:04.238140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:04.238296 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:25:04.238484 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:04.238487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:04.238537 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:04.238613 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:04.239088 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38470: BlockingRecv error: recv got EOF from 127.0.0.1:38470 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 16:25:04.498669 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:04.497399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.4.200.125:43963)
0929 16:25:04.497626 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:04.497631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:04.497655 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:25:04.497804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:04.497807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:04.497855 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:04.497929 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:04.498505 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50508: BlockingRecv error: recv got EOF from 127.0.0.1:50508 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:25:04.755133 27422 catalog_manager.cc:1259] Loaded cluster ID: 1c7f9139744c4112b2a045fe27753234
I20260929 16:25:04.755219 27422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: loading cluster ID for follower catalog manager: success
I20260929 16:25:04.756985 27422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: acquiring CA information for follower catalog manager: success
I20260929 16:25:04.758118 27422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea4cd9eb95f04482ab2662d04f3e0555: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:25:04.823882 27455 catalog_manager.cc:1259] Loaded cluster ID: 1c7f9139744c4112b2a045fe27753234
I20260929 16:25:04.823988 27455 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: loading cluster ID for follower catalog manager: success
I20260929 16:25:04.825664 27455 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: acquiring CA information for follower catalog manager: success
I20260929 16:25:04.826869 27455 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a91daa7736554efa90da93e88c625771: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:25:05.016695 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:05.012681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.4.200.125:43963)
0929 16:25:05.012977 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:05.012983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:05.013000 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:25:05.013164 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:05.013168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:05.013330 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:05.013451 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:05.013736 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.014279 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.015174 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.015424 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.015538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:05.015860 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:05.016287 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:05.016424 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:05.016465 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 16:25:05.016524 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
I20260929 16:25:05.023197 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:05.019680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.4.200.124:34211)
0929 16:25:05.019932 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:05.019936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:05.019949 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:25:05.020076 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:05.020079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:05.020182 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:05.020316 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:05.020543 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.021046 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.021823 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.022036 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.022111 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:05.022358 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:05.022813 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:05.022900 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:05.022962 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 16:25:05.023006 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260929 16:25:05.030610 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:05.026315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.4.200.126:44705)
0929 16:25:05.026573 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:05.026578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:05.026594 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:25:05.026693 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:05.026697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:05.026889 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:05.027231 (+ 342us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:05.027354 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.028146 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.029094 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.029402 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.029523 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:05.029838 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:05.030116 (+ 278us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:05.030229 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:05.030352 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:25:05.030457 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":107,"threads_started":1}
I20260929 16:25:05.031334 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:05.026028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.4.200.125:43963)
0929 16:25:05.026150 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:05.026156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:05.026684 (+ 528us) server_negotiation.cc:407] Connection header received
0929 16:25:05.026826 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:05.026830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:05.026887 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:05.026965 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:05.027222 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.027769 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.028893 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.029248 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.030263 (+ 1015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:05.030885 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:05.030971 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:05.031060 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:05.031149 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:25:05.031204 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:25:05.031672 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:05.026277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.4.200.124:34211)
0929 16:25:05.026423 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:05.026427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:05.026684 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:25:05.026885 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:05.026889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:05.026945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:05.027011 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:05.027363 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.028113 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.029198 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:05.030355 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:05.030507 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:05.031167 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:05.031241 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:05.031292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:05.031492 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 16:25:05.031547 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:25:05.034061 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27170
I20260929 16:25:05.040518 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27245
I20260929 16:25:05.046552 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27320
2026-09-29T16:25:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3154 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:05 dist-test-slave-s9ct krb5kdc[27539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:05 dist-test-slave-s9ct krb5kdc[27539](info): set up 2 sockets
Sep 29 16:25:05 dist-test-slave-s9ct krb5kdc[27539](info): commencing operation
krb5kdc: starting...
W20260929 16:25:06.082928 4897 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/test-user.keytab.
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.200.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
Entry for principal kudu/127.4.200.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab.
I20260929 16:25:06.213212 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39337
--webserver_interface=127.4.200.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:06.324035 27552 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:25:06.324389 27552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:06.324465 27552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:06.324549 27552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:06.328688 27552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:06.328779 27552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:06.328817 27552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:06.328855 27552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:06.328892 27552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:06.334220 27552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.126.keytab
--principal=kudu/127.4.200.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.126:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.200.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:06.335347 27552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:06.336407 27552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:06.345074 27560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:06.345064 27552 server_base.cc:1034] running on GCE node
W20260929 16:25:06.345060 27557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:06.345041 27558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:06.345921 27552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:06.347142 27552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:06.348312 27552 hybrid_clock.cc:648] HybridClock initialized: now 1790699106348293 us; error 33 us; skew 500 ppm
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:06.351351 27552 init.cc:377] Logged in from keytab as kudu/127.4.200.126@KRBTEST.COM (short username kudu)
I20260929 16:25:06.352304 27552 webserver.cc:460] Webserver started at http://127.4.200.126:46409/ using document root <none> and password file <none>
I20260929 16:25:06.352880 27552 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:06.352969 27552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:06.353205 27552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:06.355346 27552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data/instance:
uuid: "130aaeb80c314efc93de21148f9e354b"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "caceda36f61458a994b7a22860dff399"
server_key_iv: "6714d497cadf267730330742df1661e7"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.355964 27552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance:
uuid: "130aaeb80c314efc93de21148f9e354b"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "caceda36f61458a994b7a22860dff399"
server_key_iv: "6714d497cadf267730330742df1661e7"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.359812 27552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:06.362267 27567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:06.363539 27552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:25:06.363690 27552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal
uuid: "130aaeb80c314efc93de21148f9e354b"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "caceda36f61458a994b7a22860dff399"
server_key_iv: "6714d497cadf267730330742df1661e7"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.363816 27552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:06.393122 27552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:06.396944 27552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:06.397181 27552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:06.404969 27552 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.126:39337
I20260929 16:25:06.405292 27619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.126:39337 every 8 connection(s)
I20260929 16:25:06.406018 27552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:06.408877 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:06.410373 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27552
I20260929 16:25:06.410547 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-0/wal/instance
I20260929 16:25:06.410861 4897 external_mini_cluster.cc:1428] Setting key e0e4f01cdc3e7283be9d88024af5d9b3
I20260929 16:25:06.413903 27620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.200.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.125@KRBTEST.COM" created.
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:06.430420 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.417263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.4.200.126:39337)
0929 16:25:06.417617 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.417633 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.417687 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:25:06.418194 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.418201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.418534 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.418774 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.419726 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.420720 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.421845 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.422445 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.422629 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.425125 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.425135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.425151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.425181 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.427253 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.427777 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.427786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.427791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.427941 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.428261 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.428267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.428270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.428855 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.428992 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.429366 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.429592 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260929 16:25:06.430449 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.416349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39337 (local address 127.0.0.1:46040)
0929 16:25:06.416854 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.416907 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.417855 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.418777 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.418782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.419137 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.419561 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.419578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.420856 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.420859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.421698 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.421708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.422848 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.422852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.422926 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.423523 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.423540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.424975 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.427373 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.427382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.427390 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.427678 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.428052 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.428054 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.428055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.428166 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.428971 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.428977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.429102 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.429239 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.429506 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":43,"thread_start_us":147,"threads_started":1}
I20260929 16:25:06.433068 27620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.433837 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.433578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56320)
0929 16:25:06.433692 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.433771 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:25:06.434077 27568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
I20260929 16:25:06.435001 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.434791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56326)
0929 16:25:06.434898 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.434946 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:25:06.435171 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:32961: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
Entry for principal kudu/127.4.200.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab.
I20260929 16:25:06.442018 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:32961
--webserver_interface=127.4.200.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:25:06.484462 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } attempt: 1
I20260929 16:25:06.485354 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.485018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56342)
0929 16:25:06.485189 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.485261 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:25:06.486135 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.485907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56356)
0929 16:25:06.486020 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.486072 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:25:06.486266 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:32961: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
W20260929 16:25:06.552563 27627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:25:06.552806 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:06.552870 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:06.552939 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:25:06.554455 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } attempt: 2
I20260929 16:25:06.555326 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.555026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56366)
0929 16:25:06.555185 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.555251 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:25:06.556041 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.555805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56380)
0929 16:25:06.555926 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.555979 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:25:06.556190 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.125:32961: Network error: Client connection negotiation failed: client connection to 127.4.200.125:32961: connect: Connection refused (error 111)
W20260929 16:25:06.557614 27627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:06.557664 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:06.557684 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:06.557701 27627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:06.557716 27627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:06.563381 27627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.125.keytab
--principal=kudu/127.4.200.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.125:32961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.200.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:06.564399 27627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:06.565431 27627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:06.574254 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:06.574254 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:06.574249 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:06.574693 27627 server_base.cc:1034] running on GCE node
I20260929 16:25:06.575333 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:06.576440 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:06.577617 27627 hybrid_clock.cc:648] HybridClock initialized: now 1790699106577585 us; error 43 us; skew 500 ppm
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:06.580621 27627 init.cc:377] Logged in from keytab as kudu/127.4.200.125@KRBTEST.COM (short username kudu)
I20260929 16:25:06.581609 27627 webserver.cc:460] Webserver started at http://127.4.200.125:38123/ using document root <none> and password file <none>
I20260929 16:25:06.582212 27627 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:06.582307 27627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:06.582578 27627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:06.584705 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data/instance:
uuid: "42fee472658f4557a208338136ba093c"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "cd8d5cd13d90abf7123b95716073014a"
server_key_iv: "8dc6cb3468b0990e966bab8729222ced"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.585278 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal/instance:
uuid: "42fee472658f4557a208338136ba093c"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "cd8d5cd13d90abf7123b95716073014a"
server_key_iv: "8dc6cb3468b0990e966bab8729222ced"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.589126 27627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:25:06.591539 27642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:06.592777 27627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:25:06.592927 27627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal
uuid: "42fee472658f4557a208338136ba093c"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "cd8d5cd13d90abf7123b95716073014a"
server_key_iv: "8dc6cb3468b0990e966bab8729222ced"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.593041 27627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:06.624215 27627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:06.627779 27627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:06.628062 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:06.635461 27627 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.125:32961
I20260929 16:25:06.635748 27694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.125:32961 every 8 connection(s)
I20260929 16:25:06.636489 27627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/data/info.pb
I20260929 16:25:06.639096 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27627
I20260929 16:25:06.639277 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-1/wal/instance
I20260929 16:25:06.639583 27695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:06.639636 4897 external_mini_cluster.cc:1428] Setting key e7a776fb17ba81dd3811bf5b4a592b60
I20260929 16:25:06.643833 27695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.200.124@KRBTEST.COM; defaulting to no policy
I20260929 16:25:06.647383 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.200.124@KRBTEST.COM" created.
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:06.659152 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.647836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56390)
0929 16:25:06.648024 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.648042 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.648109 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.648770 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.648775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.648938 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.649040 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.649046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.650502 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.650506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.651194 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.651204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.652069 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.652072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.652165 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.652758 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.652777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.654309 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.656866 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.656871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.656876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.657380 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.657843 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.657846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.657848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.657933 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.658659 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.658662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.658759 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.658863 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.658992 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:25:06.660104 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.645711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.4.200.126:39337)
0929 16:25:06.645901 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.645910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.646675 (+ 765us) server_negotiation.cc:407] Connection header received
0929 16:25:06.647700 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.647703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.647758 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.647844 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.649376 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.650028 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.651451 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.651805 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.651928 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.655254 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.655259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.655262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.655291 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.657290 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.657749 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.657753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.657754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.657865 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.658260 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.658263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.658265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.658487 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.658591 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.659766 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.659898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:25:06.660238 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.647928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.4.200.125:32961)
0929 16:25:06.648267 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.648279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.648305 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:25:06.648351 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.648356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.648546 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.648740 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.649175 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.650349 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.651323 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.651935 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.652118 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.654490 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.654500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.654515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.654546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.656735 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.657544 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.657552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.657556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.657730 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.658030 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.658033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.658035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.658544 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.658691 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.658971 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.659261 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":378,"server-negotiator.queue_time_us":242,"thread_start_us":134,"threads_started":1}
I20260929 16:25:06.660750 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.646012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39337 (local address 127.0.0.1:46046)
0929 16:25:06.646497 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.646548 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.647436 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.648039 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.648046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.648532 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.649214 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.649229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.650154 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.650158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.651305 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.651322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.651988 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.651996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.652234 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.652979 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.652995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.655040 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.657397 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.657408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.657417 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.657634 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.657973 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.657978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.657980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.658133 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.658602 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.658609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.658792 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.660323 (+ 1531us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.660536 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":8,"thread_start_us":146,"threads_started":1}
I20260929 16:25:06.661948 27695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.662360 27620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.663028 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.662803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51056)
0929 16:25:06.662908 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.662961 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:25:06.663827 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.663562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51060)
0929 16:25:06.663736 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.663779 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 16:25:06.663990 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
Entry for principal kudu/127.4.200.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab.
I20260929 16:25:06.669940 4897 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
/tmp/dist-test-taskh_zHSI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:44897
--webserver_interface=127.4.200.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.200.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:25:06.671859 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.662480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56392)
0929 16:25:06.662898 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.662911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.662993 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.663244 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.663247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.663421 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.663545 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.663553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.664665 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.664669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.665302 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.665311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.665848 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.665851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.665925 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.666450 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.666466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.667898 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.670016 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.670019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.670020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.670336 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.670728 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.670731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.670732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.670813 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.671311 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.671316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.671413 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.671527 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.671665 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350}
I20260929 16:25:06.671859 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.662640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.4.200.125:32961)
0929 16:25:06.662769 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.662774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.663073 (+ 299us) server_negotiation.cc:407] Connection header received
0929 16:25:06.663104 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.663108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.663142 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.663208 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.663967 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.664562 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.665466 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.665744 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.665827 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.668037 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.668040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.668042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.668062 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.669887 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.670520 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.670526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.670530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.670620 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.670914 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.670920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.670925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.671162 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.671269 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.671584 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.671720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":66}
I20260929 16:25:06.672559 27695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.673223 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.672992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51068)
0929 16:25:06.673096 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.673162 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:25:06.673786 27646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
I20260929 16:25:06.674762 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.674556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51078)
0929 16:25:06.674657 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.674715 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:25:06.674917 27695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
I20260929 16:25:06.695226 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 1
I20260929 16:25:06.696092 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.695788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51082)
0929 16:25:06.695935 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.696005 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:25:06.696836 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.696611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51092)
0929 16:25:06.696730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.696782 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:25:06.697018 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
I20260929 16:25:06.724170 27695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 1
I20260929 16:25:06.725066 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.724776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51102)
0929 16:25:06.724926 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.724987 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:25:06.725807 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.725590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51104)
0929 16:25:06.725705 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.725754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:25:06.725943 27695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
I20260929 16:25:06.772200 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 2
I20260929 16:25:06.773067 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.772749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51110)
0929 16:25:06.772904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.772983 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:25:06.773895 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.773677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51118)
0929 16:25:06.773784 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.773838 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:25:06.774034 27620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
W20260929 16:25:06.781718 27701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:25:06.782007 27701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:06.782068 27701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:06.782142 27701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:06.786142 27701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:06.786221 27701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:06.786268 27701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:06.786305 27701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:06.786339 27701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:06.791428 27701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.200.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.200.126:39337,127.4.200.125:32961,127.4.200.124:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/krb5kdc/kudu_127.4.200.124.keytab
--principal=kudu/127.4.200.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.200.124:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.200.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:06.792421 27701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:06.793366 27701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:25:06.794133 27695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 2
I20260929 16:25:06.795176 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.794827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51122)
0929 16:25:06.795022 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.795096 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:25:06.796032 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.795810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51132)
0929 16:25:06.795923 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.795975 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:25:06.796160 27695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.200.124:44897: Network error: Client connection negotiation failed: client connection to 127.4.200.124:44897: connect: Connection refused (error 111)
W20260929 16:25:06.802611 27707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:06.802578 27710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:06.802584 27708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:06.802883 27701 server_base.cc:1034] running on GCE node
I20260929 16:25:06.803617 27701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:06.804719 27701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:06.805905 27701 hybrid_clock.cc:648] HybridClock initialized: now 1790699106805872 us; error 48 us; skew 500 ppm
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:06.808521 27701 init.cc:377] Logged in from keytab as kudu/127.4.200.124@KRBTEST.COM (short username kudu)
I20260929 16:25:06.809458 27701 webserver.cc:460] Webserver started at http://127.4.200.124:43577/ using document root <none> and password file <none>
I20260929 16:25:06.810043 27701 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:06.810134 27701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:06.810374 27701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:06.812480 27701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data/instance:
uuid: "47a1db69f63f4e83ab9838f49c2a5568"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "5b951a6edd1e6c6d30f71bd9b5bdd7c0"
server_key_iv: "50f1a226726b10e904d013bc9086cc9a"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.813064 27701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal/instance:
uuid: "47a1db69f63f4e83ab9838f49c2a5568"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "5b951a6edd1e6c6d30f71bd9b5bdd7c0"
server_key_iv: "50f1a226726b10e904d013bc9086cc9a"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.816887 27701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:25:06.819490 27717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:06.820744 27701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:25:06.820904 27701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data,/tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal
uuid: "47a1db69f63f4e83ab9838f49c2a5568"
format_stamp: "Formatted at 2026-09-29 16:25:06 on dist-test-slave-s9ct"
server_key: "5b951a6edd1e6c6d30f71bd9b5bdd7c0"
server_key_iv: "50f1a226726b10e904d013bc9086cc9a"
server_key_version: "encryptionkey@0"
I20260929 16:25:06.821018 27701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:06.832715 27701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:06.835917 27701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:06.836146 27701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:06.843420 27701 rpc_server.cc:307] RPC server started. Bound to: 127.4.200.124:44897
I20260929 16:25:06.843720 27769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.200.124:44897 every 8 connection(s)
I20260929 16:25:06.844463 27701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/data/info.pb
I20260929 16:25:06.846659 4897 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu as pid 27701
I20260929 16:25:06.846824 4897 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh_zHSI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790698821455595-4897-0/minicluster-data/master-2/wal/instance
I20260929 16:25:06.847142 4897 external_mini_cluster.cc:1428] Setting key 71bf3044f73446471add31f39f97fdea
I20260929 16:25:06.847707 27770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:06.852330 27770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
I20260929 16:25:06.867328 27773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.848499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.4.200.124:44897)
0929 16:25:06.848909 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.848938 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.848975 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:25:06.849638 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.849654 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.849998 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.850288 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.851285 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.852307 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.853469 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.854008 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.854181 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.857301 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.857307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.857316 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.857343 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.859535 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.860263 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.860270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.860575 (+ 305us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.860731 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.864912 (+ 4181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.864919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.864924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.865455 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.865617 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.865985 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.866230 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260929 16:25:06.868285 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.854260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.126:39337 (local address 127.0.0.1:46052)
0929 16:25:06.854745 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.854795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.854913 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.855290 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.855295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.855623 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.855744 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.855749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.856710 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.856733 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.857435 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.857445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.857961 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.857965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.858093 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.858687 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.858700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.860215 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.866449 (+ 6234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.866459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.866465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.866743 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.867049 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.867052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.867053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.867128 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.867712 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.867725 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.867847 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.867985 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.868126 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":241,"threads_started":1}
I20260929 16:25:06.868383 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.854378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.4.200.126:39337)
0929 16:25:06.854568 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.854576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.854882 (+ 306us) server_negotiation.cc:407] Connection header received
0929 16:25:06.855031 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.855035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.855104 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.855221 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.855903 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.856545 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.857581 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.857830 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.857959 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.860666 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.860671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.860674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.860703 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.866281 (+ 5578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.866851 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.866855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.866856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.866950 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.867255 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.867258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.867260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.867510 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.867697 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.868098 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.868217 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:25:06.869554 27770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.873201 27620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 3
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.126@KRBTEST.COM
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.126@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:25:06.879494 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.870155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.125:32961 (local address 127.0.0.1:56406)
0929 16:25:06.870331 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.870346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.870444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.870679 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.870681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.870859 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.870948 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.870953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.871676 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.871679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.872194 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.872201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.872681 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.872684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.872795 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.873389 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.873405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.875513 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.877740 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.877744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.877748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.877989 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.878303 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.878306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.878308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.878435 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.878863 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.878866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.879044 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.879177 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.879318 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 16:25:06.879657 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.870247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.4.200.125:32961)
0929 16:25:06.870420 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.870425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.870440 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:25:06.870528 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.870530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.870573 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.870666 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.871065 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.871560 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.872312 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.872565 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.872680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.875715 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.875719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.875722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.875753 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.877571 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.878102 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.878106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.878107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.878193 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.878530 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.878534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.878536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.878762 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.878875 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.879270 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.879481 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:25:06.880821 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.869905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.4.200.126:39337)
0929 16:25:06.870026 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.870030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.870090 (+ 60us) server_negotiation.cc:407] Connection header received
0929 16:25:06.870242 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.870245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.870301 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.870375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.871010 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.871513 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.872313 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.872543 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.872653 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.876312 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.876315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.876317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.876339 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.878293 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.878976 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.878979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.878981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.879062 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.879377 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.879380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.879381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.879613 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.879845 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.880520 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.880642 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:25:06.880795 27770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:25:06.883101 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.873689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51150)
0929 16:25:06.873874 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.873891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.873969 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.874249 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.874255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.874503 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.874604 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.874613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.875679 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.875683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.876509 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.876524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.877145 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.877148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.877232 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.877668 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.877682 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.879257 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.881479 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.881482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.881486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.881722 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.882032 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.882035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.882038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.882133 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.882607 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.882611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.882705 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.882815 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.882945 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:25:06.883111 27773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.873766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.4.200.124:44897)
0929 16:25:06.873906 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.873911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.873930 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:25:06.874064 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.874069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.874124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.874220 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.874745 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.875521 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.876636 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.877023 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.877146 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.879405 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.879410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.879416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.879442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.881071 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.881846 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.881849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.881851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.881927 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.882243 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.882246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.882248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.882495 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.882603 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.882873 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.882976 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":51}
I20260929 16:25:06.886255 27620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: Bootstrap starting.
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.200.125@KRBTEST.COM
I20260929 16:25:06.887327 27695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } attempt: 3
I20260929 16:25:06.888949 27620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:06.889869 27620 log.cc:824] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.124@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:25:06.892258 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.881980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.4.200.125:32961)
0929 16:25:06.882083 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.882087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.882273 (+ 186us) server_negotiation.cc:407] Connection header received
0929 16:25:06.882310 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.882313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.882358 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.883350 (+ 992us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.883445 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.884175 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.884904 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.885213 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.885327 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.887840 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.887844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.887846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.887871 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.890031 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.890554 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.890556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.890558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.890638 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.890959 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.890962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.890966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.891177 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.891873 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.891944 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.892044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:25:06.892627 27620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: No bootstrap required, opened a new log
Sep 29 16:25:06 dist-test-slave-s9ct krb5kdc[27539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.200.125@KRBTEST.COM for kudu/127.4.200.124@KRBTEST.COM
I20260929 16:25:06.895759 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.881373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.4.200.124:44897)
0929 16:25:06.881960 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.881975 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.881989 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:25:06.882022 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.882025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.882066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.882151 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.882672 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.886953 (+ 4281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.887966 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.888287 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.888381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.890681 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.890685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.890689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.890715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.893477 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.893918 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.893921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.893922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.894013 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.894774 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.894778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.894780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.894997 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.895084 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.895419 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.895554 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260929 16:25:06.895794 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.881325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51160)
0929 16:25:06.881422 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.881436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.881513 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.882169 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.882172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.882371 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.882559 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.882566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.887071 (+ 4505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.887075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.887842 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.887852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.888474 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.888477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.888545 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.889118 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.889132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.890559 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.893575 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.893578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.893579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.893826 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.894120 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.894123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.894125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.894207 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.895151 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.895153 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.895242 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.895494 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.895622 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":41}
I20260929 16:25:06.896522 4897 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:25:06.897423 27620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.897672 27620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130aaeb80c314efc93de21148f9e354b, State: Initialized, Role: FOLLOWER
I20260929 16:25:06.897635 27588 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "130aaeb80c314efc93de21148f9e354b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46068
I20260929 16:25:06.898072 27620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.898676 27777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.898818 27777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.899036 27770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: Bootstrap starting.
I20260929 16:25:06.899410 27620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:06.899442 27588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:25:06.899582 27588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:06.900774 27588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.901726 27588 leader_election.cc:288] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [CANDIDATE]: Term 1 election: Requested vote from peers 42fee472658f4557a208338136ba093c (127.4.200.125:32961), 47a1db69f63f4e83ab9838f49c2a5568 (127.4.200.124:44897)
I20260929 16:25:06.901917 27770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:06.902931 27770 log.cc:824] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:06.906032 27770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: No bootstrap required, opened a new log
I20260929 16:25:06.906723 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.887708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.200.124:44897 (local address 127.0.0.1:51174)
0929 16:25:06.887948 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:06.887962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:06.888032 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:06.889396 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:06.889399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:06.889620 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:06.889729 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.889737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.891250 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.891255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.892009 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:06.892020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.893047 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.893051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:06.893129 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.893551 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:06.893565 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:06.895139 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:06.904912 (+ 9773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.904916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.904920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.905185 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.905538 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:06.905541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:06.905543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:06.905643 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:06.906135 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:06.906140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:06.906275 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 16:25:06.906367 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:25:06.906550 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1}
I20260929 16:25:06.907220 27773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.888426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.4.200.124:44897)
0929 16:25:06.889175 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.889179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.889192 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:25:06.889226 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.889229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.889265 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.889484 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.890497 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.891151 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.892132 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.892905 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.895277 (+ 2372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.895377 (+ 100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.895380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.895383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.895399 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.904778 (+ 9379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.905316 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.905320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.905323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.905427 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.905762 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.905766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.905767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.905997 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.906885 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.906959 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.907069 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 16:25:06.904676 27552 master_runner.cc:186] Error getting master registration for 127.4.200.125:32961: OK, instance_id { permanent_uuid: "42fee472658f4557a208338136ba093c" instance_seqno: 1790699106633451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:25:06.908294 27552 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:25:06.909876 27770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.910120 27770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a1db69f63f4e83ab9838f49c2a5568, State: Initialized, Role: FOLLOWER
I20260929 16:25:06.910487 27770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.911326 27695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: Bootstrap starting.
W20260929 16:25:06.911432 27788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:06.911496 27788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:06.911760 27770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:06.913972 27695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:06.913915 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.914052 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.914884 27695 log.cc:824] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:06.915328 27744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "130aaeb80c314efc93de21148f9e354b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "47a1db69f63f4e83ab9838f49c2a5568"
I20260929 16:25:06.915545 27744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:06.916823 27744 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 130aaeb80c314efc93de21148f9e354b in term 1.
I20260929 16:25:06.917351 27571 leader_election.cc:302] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130aaeb80c314efc93de21148f9e354b, 47a1db69f63f4e83ab9838f49c2a5568; no voters:
W20260929 16:25:06.917513 27809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:06.917587 27809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:06.917683 27777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:06.917940 27777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [term 1 LEADER]: Becoming Leader. State: Replica: 130aaeb80c314efc93de21148f9e354b, State: Running, Role: LEADER
I20260929 16:25:06.918246 27777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.920174 27701 master_runner.cc:186] Error getting master registration for 127.4.200.125:32961: OK, instance_id { permanent_uuid: "42fee472658f4557a208338136ba093c" instance_seqno: 1790699106633451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:25:06.922883 27701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:25:06.923333 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 130aaeb80c314efc93de21148f9e354b. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.923442 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:06.923856 27814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:06.924808 27814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:06.931282 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.904348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.4.200.124:44897)
0929 16:25:06.904454 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.904459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.911794 (+ 7335us) server_negotiation.cc:407] Connection header received
0929 16:25:06.911983 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.911987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.912028 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.912073 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.913267 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.914049 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.924757 (+ 10708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.925095 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.925176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.927380 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.927384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.927387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.927409 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.929392 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.929868 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.929872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.929874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.929961 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.930254 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.930258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.930259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.930495 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.930625 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.930979 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.931119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:25:06.932499 27695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: No bootstrap required, opened a new log
I20260929 16:25:06.935684 27744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [term 1 FOLLOWER]: Refusing update from remote peer 130aaeb80c314efc93de21148f9e354b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:25:06.936796 27812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:25:06.941203 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 130aaeb80c314efc93de21148f9e354b. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.941355 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.943213 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.904885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.4.200.125:32961)
0929 16:25:06.904982 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.904986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.926225 (+ 21239us) server_negotiation.cc:407] Connection header received
0929 16:25:06.927628 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.927634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.927700 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.927754 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.928680 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.929466 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.935429 (+ 5963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.935738 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.935816 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.938187 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.938192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.938195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.938223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.939857 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.940681 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.940685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.940687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.940774 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.941529 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.941532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.941534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.941757 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.941877 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.942905 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.943036 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:25:06.943742 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:06.904299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.4.200.126:39337)
0929 16:25:06.904424 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:06.904431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:06.925988 (+ 21557us) server_negotiation.cc:407] Connection header received
0929 16:25:06.927908 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:06.927913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:06.927963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:06.928037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:06.932364 (+ 4327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.933103 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.934134 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:06.934505 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:06.934612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:06.937253 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:06.937257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:06.937260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:06.937288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:06.939392 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.941101 (+ 1709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.941105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.941106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.941210 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:06.942201 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:06.942204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:06.942206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:06.942452 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:06.942553 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:06.943456 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 16:25:06.943576 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:25:06.945716 27695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.945938 27695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fee472658f4557a208338136ba093c, State: Initialized, Role: FOLLOWER
I20260929 16:25:06.946269 27695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } }
I20260929 16:25:06.946806 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.947032 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.948441 27695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:06.948729 27668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:06.950210 27668 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [term 1 FOLLOWER]: Refusing update from remote peer 130aaeb80c314efc93de21148f9e354b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:25:06.950825 27812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:25:06.950670 27669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "130aaeb80c314efc93de21148f9e354b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "42fee472658f4557a208338136ba093c"
I20260929 16:25:06.952215 27669 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 130aaeb80c314efc93de21148f9e354b in term 1.
I20260929 16:25:06.955546 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 130aaeb80c314efc93de21148f9e354b. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.955659 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.957532 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.957645 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: This master's current role is: LEADER
W20260929 16:25:06.958313 27829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:06.958375 27829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:06.958773 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.958881 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.959446 27815 mvcc.cc:205] Tried to move back new op lower bound from 7334703541986025472 to 7334703541940662272. Current Snapshot: MvccSnapshot[applied={T|T < 7334703541986025472}]
I20260929 16:25:06.959632 27816 mvcc.cc:205] Tried to move back new op lower bound from 7334703541986025472 to 7334703541940662272. Current Snapshot: MvccSnapshot[applied={T|T < 7334703541986025472}]
I20260929 16:25:06.961930 27777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.962040 27777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:06.962598 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "130aaeb80c314efc93de21148f9e354b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130aaeb80c314efc93de21148f9e354b" member_type: VOTER last_known_addr { host: "127.4.200.126" port: 39337 } } peers { permanent_uuid: "42fee472658f4557a208338136ba093c" member_type: VOTER last_known_addr { host: "127.4.200.125" port: 32961 } } peers { permanent_uuid: "47a1db69f63f4e83ab9838f49c2a5568" member_type: VOTER last_known_addr { host: "127.4.200.124" port: 44897 } } }
I20260929 16:25:06.962700 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:25:06.967995 27814 catalog_manager.cc:1347] Generated new cluster ID: 0b1b22ebf8ed448b9dd5e85aa51eed21
I20260929 16:25:06.968065 27814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:06.968751 27830 mvcc.cc:205] Tried to move back new op lower bound from 7334703541986025472 to 7334703541940662272. Current Snapshot: MvccSnapshot[applied={T|T < 7334703541986025472}]
I20260929 16:25:07.010212 27814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:07.011540 27814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:07.029474 27814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 130aaeb80c314efc93de21148f9e354b: Generated new TSK 0
I20260929 16:25:07.030326 27814 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:25:07.061715 27775 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.059829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.4.200.124:44897)
0929 16:25:07.059957 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.059962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.060607 (+ 645us) server_negotiation.cc:407] Connection header received
0929 16:25:07.060814 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.060818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.060868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.060921 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.061575 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51194: BlockingRecv error: recv got EOF from 127.0.0.1:51194 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:25:07.064278 27624 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.062593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.4.200.126:39337)
0929 16:25:07.062706 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.062711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.063390 (+ 679us) server_negotiation.cc:407] Connection header received
0929 16:25:07.063561 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.063564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.063612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.063662 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.064149 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46088: BlockingRecv error: recv got EOF from 127.0.0.1:46088 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:25:07.065899 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.063105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.4.200.125:32961)
0929 16:25:07.063235 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.063239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.064997 (+ 1758us) server_negotiation.cc:407] Connection header received
0929 16:25:07.065165 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.065169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.065227 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.065277 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.065772 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56434: BlockingRecv error: recv got EOF from 127.0.0.1:56434 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:25:07.070243 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.068794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.4.200.125:32961)
0929 16:25:07.068892 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.068896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.069307 (+ 411us) server_negotiation.cc:407] Connection header received
0929 16:25:07.069524 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.069527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.069601 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.069654 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.070131 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56440: BlockingRecv error: recv got EOF from 127.0.0.1:56440 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 16:25:07.075564 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.073940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.4.200.125:32961)
0929 16:25:07.074052 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.074056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.074630 (+ 574us) server_negotiation.cc:407] Connection header received
0929 16:25:07.074826 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.074829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.074879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.074928 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.075452 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56444: BlockingRecv error: recv got EOF from 127.0.0.1:56444 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:25:07.081777 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.080155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.4.200.125:32961)
0929 16:25:07.080281 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.080286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.080803 (+ 517us) server_negotiation.cc:407] Connection header received
0929 16:25:07.081006 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.081009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.081063 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.081113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.081659 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56446: BlockingRecv error: recv got EOF from 127.0.0.1:56446 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 16:25:07.091354 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.089401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.4.200.125:32961)
0929 16:25:07.089584 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.089591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.090221 (+ 630us) server_negotiation.cc:407] Connection header received
0929 16:25:07.090474 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.090479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.090540 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.090597 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.091221 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56452: BlockingRecv error: recv got EOF from 127.0.0.1:56452 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260929 16:25:07.104444 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.102600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.4.200.125:32961)
0929 16:25:07.102729 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.102734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.103333 (+ 599us) server_negotiation.cc:407] Connection header received
0929 16:25:07.103571 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.103576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.103633 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.103689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.104296 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56458: BlockingRecv error: recv got EOF from 127.0.0.1:56458 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 16:25:07.125227 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.123444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.4.200.125:32961)
0929 16:25:07.123570 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.123574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.124192 (+ 618us) server_negotiation.cc:407] Connection header received
0929 16:25:07.124423 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.124427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.124481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.124534 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.125087 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56470: BlockingRecv error: recv got EOF from 127.0.0.1:56470 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:25:07.161060 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.159938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.4.200.125:32961)
0929 16:25:07.160090 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.160096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.160200 (+ 104us) server_negotiation.cc:407] Connection header received
0929 16:25:07.160315 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.160318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.160381 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.160497 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.160922 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56486: BlockingRecv error: recv got EOF from 127.0.0.1:56486 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:25:07.228609 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.227369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.4.200.125:32961)
0929 16:25:07.227540 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.227545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.227663 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:25:07.227846 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.227850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.227909 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.228004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.228468 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56498: BlockingRecv error: recv got EOF from 127.0.0.1:56498 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 16:25:07.360458 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.359268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.4.200.125:32961)
0929 16:25:07.359438 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.359442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.359682 (+ 240us) server_negotiation.cc:407] Connection header received
0929 16:25:07.359794 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.359797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.359849 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.359925 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.360356 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56504: BlockingRecv error: recv got EOF from 127.0.0.1:56504 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:25:07.619911 27699 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:25:07.618781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.4.200.125:32961)
0929 16:25:07.618921 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:07.618925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:07.619045 (+ 120us) server_negotiation.cc:407] Connection header received
0929 16:25:07.619171 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:07.619174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:07.619224 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:07.619309 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:07.619768 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56506: BlockingRecv error: recv got EOF from 127.0.0.1:56506 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:25:07.918921 27809 catalog_manager.cc:1259] Loaded cluster ID: 0b1b22ebf8ed448b9dd5e85aa51eed21
I20260929 16:25:07.919013 27809 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: loading cluster ID for follower catalog manager: success
I20260929 16:25:07.920663 27809 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: acquiring CA information for follower catalog manager: success
I20260929 16:25:07.921779 27809 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47a1db69f63f4e83ab9838f49c2a5568: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:25:07.959775 27829 catalog_manager.cc:1259] Loaded cluster ID: 0b1b22ebf8ed448b9dd5e85aa51eed21
I20260929 16:25:07.959860 27829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: loading cluster ID for follower catalog manager: success
I20260929 16:25:07.961447 27829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: acquiring CA information for follower catalog manager: success
I20260929 16:25:07.962556 27829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42fee472658f4557a208338136ba093c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:25:08.138371 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:08.134419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.4.200.125:32961)
0929 16:25:08.134744 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:08.134749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:08.134845 (+ 96us) server_negotiation.cc:407] Connection header received
0929 16:25:08.135030 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:08.135033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:08.135204 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:08.135346 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:08.135629 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.136164 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.137016 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.137232 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.137307 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:08.137581 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:08.138037 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:08.138128 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:08.138193 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 16:25:08.138242 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":127,"threads_started":1}
I20260929 16:25:08.145151 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:08.141371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.4.200.124:44897)
0929 16:25:08.141618 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:08.141623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:08.141639 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:25:08.141732 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:08.141735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:08.141852 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:08.141981 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:08.142197 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.142780 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.143605 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.143839 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.143945 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:08.144252 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:08.144749 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:08.144898 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:08.144939 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 16:25:08.144992 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260929 16:25:08.152071 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:08.148098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.4.200.124:44897)
0929 16:25:08.148471 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:08.148474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:08.148486 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:25:08.148521 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:08.148524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:08.148567 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:08.148633 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:08.148866 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.149609 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.150960 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.151259 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.151376 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:08.151611 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:08.151686 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:08.151763 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:08.151864 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 16:25:08.151911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260929 16:25:08.152488 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:08.148449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.4.200.125:32961)
0929 16:25:08.148578 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:08.148581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:08.148918 (+ 337us) server_negotiation.cc:407] Connection header received
0929 16:25:08.149023 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:08.149026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:08.149078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:08.149179 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:08.149452 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.150117 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.151153 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.151479 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.151571 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:08.152015 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:08.152086 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:08.152136 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:08.152292 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:25:08.152341 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:25:08.153457 27911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:08.147920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.4.200.126:39337)
0929 16:25:08.148297 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:08.148303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:08.148322 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:25:08.148534 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:08.148537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:08.148692 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:08.148817 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:25:08.150012 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.150850 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.151996 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:08.152355 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:08.152448 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:08.152765 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:25:08.153098 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:25:08.153171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:08.153308 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:25:08.153354 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":62,"threads_started":1}
I20260929 16:25:08.156275 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27552
I20260929 16:25:08.162899 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27627
I20260929 16:25:08.168754 4897 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh_zHSI/build/debug/bin/kudu with pid 27701
2026-09-29T16:25:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3122 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6276 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286705 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST